Fix TestDelayedEvents/delayed_state_events_are_kept_on_server_restart to account for slow servers (de-flake)

[MadLittleMods]

Follow-up to #829

Part of element-hq/synapse#18537

What does this PR do?

Fix TestDelayedEvents/delayed_state_events_are_kept_on_server_restart to account for slow servers (de-flake).

Previously, this was naively using a single delayed event with a 10 second delay. But because we're stopping and starting servers here, it could take up deployment.GetConfig().SpawnHSTimeout (defaults to 30 seconds) for the server to start up again so by the time the server is back up, the delayed event may have already been sent, invalidating our assertions below (which expect some delayed events to still be pending and then see one of them be sent after the server is back up).

We could account for this by setting the delayed event delay to be longer than deployment.GetConfig().SpawnHSTimeout but that would make the test suite take longer to run in all cases even for homeservers that are quick to restart because we have to wait for that large delay.

We instead account for this by scheduling many delayed events at short intervals (we chose 10 seconds because that's what the test naively chose before). Then whenever the servers comes back, we can just check until it decrements by 1.

Experiencing the flaky failure

As experienced when running this test against the worker-based Synapse setup we use alongside the Synapse Pro Rust apps, https://github.com/element-hq/synapse-rust-apps/pull/344#discussion_r2620714306. We probably experience this heavily in the private project because GitHub runners are less than half as powerful as those for public projects and that single container with a share of the 2 CPU cores available is just not powerful enough to run all 16 workers effectively.

For reference, the CI runners provided by GitHub for private projects are less than half as powerful as those for public projects.

Standard GitHub-hosted runners for public repositories

Virtual machine	Processor (CPU)	Memory (RAM)	Storage (SSD)	Architecture	Workflow label
Linux	4	16 GB	14 GB	x64	ubuntu-latest, ubuntu-24.04, ubuntu-22.04

-- Standard GitHub-hosted runners for public repositories

---

Standard GitHub-hosted runners for private repositories

Virtual Machine	Processor (CPU)	Memory (RAM)	Storage (SSD)	Architecture	Workflow label
Linux	2	7 GB	14 GB	x64	ubuntu-latest, ubuntu-24.04, ubuntu-22.04

-- Standard GitHub-hosted runners for private repositories

And for the same slow reasons, why we're also experiencing this as an occasional flake with (workers, postgres) in the public Synapse CI as well.

Reproduction instructions

I can easily reproduce this problem if I use #827 to limit the number of CPU's available for the homeserver containers to use: COMPLEMENT_CONTAINER_CPUS=0.5

Pull Request Checklist

• Pull request includes a sign off

[MadLittleMods]

Because I needed some less than/greater than comparisons in the refactored test logic, I decided to make matchDelayedEvents like the MustSyncUntil where we can pass in a bevy of check opts.

[MadLittleMods]

Ideally, the other delayedEventsNumberGreaterThan/delayedEventsNumberLessThan checks would also use the JSON matching like we do here so we get maximum nice debug output when things go wrong. But there isn't a matcher for that yet. We could add one but I've let it be for now. Things fit nice enough as-is.

[devonh]

This looks well thought through.
Thanks for taking this on to reduce the overall flakiness of tests.

[MadLittleMods]

Thanks for the review @devonh 🐳