runner-select: 404 with backtrace

[jschwe]

POST https://ci0.servo.org/queue/enqueue?unique_id=c06fb267-ddfd-417a-9c9f-779079b1fbce&qualified_repo=servo/servo&run_id=19885823270

POST https://ci0.servo.org/queue/take/c06fb267-ddfd-417a-9c9f-779079b1fbce
curl: (22) The requested URL returned error: 500
curl: (22) The requested URL returned error: 404

   0: error sending request for url (https://ci1.servo.org/profile/servo-ubuntu2204/take?unique_id=c06fb267-ddfd-417a-9c9f-779079b1fbce&qualified_repo=servo%2Fservo&run_id=19885823270)
   1: operation timed out

Location:
   src/bin/queue.rs:482

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
   0: Not found: UniqueId("c06fb267-ddfd-417a-9c9f-779079b1fbce")

Location:
   src/bin/queue.rs:224

https://github.com/servo/servo/actions/runs/19885823270/job/56992991343

I'm not sure if this is an expected error, or if this indicates a bug in the monitor.

[jschwe]

CC @delan

[delan]

it looks like the queue timed out forwarding a reservation request to one of the monitors, which caused the queue to respond with HTTP 500. curl considers this retryable, so it retries, but now the queue has forgotten about that job, so it responds with HTTP 404.

i think it’s good that the queue ultimately rejects the retry in this case, because reservation requests might still have side effects even if they time out, but the error output could definitely be improved.

to avoid retrying and getting HTTP 404, we need to make the first response not be HTTP {408,429,500,502,503,504}. very few other status codes are appropriate since many of them are tied to specific HTTP features, but maybe we can use something like HTTP 400 or 422.