test failed in CI: test_multicast_group_dpd_communication_failure_recovery

[iximeow]

This test failed on a CI run on #9726:

https://github.com/oxidecomputer/omicron/pull/9726/checks?check_run_id=61532981157

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KFY6CT3MJF8JM8EN06V1YM9P/cl4dgMztjlUA5dUu6PW7VyiY9NUyD82axBUG9IGtLURWEj8f/01KFY6D5X9DJXSA0V9RCPYEKTA#S8412

Excerpt from the log showing the failure:

8413  2026-01-27T00:35:56.322Z
8414  2026-01-27T00:35:56.322Z      running 1 test
8415  2026-01-27T00:35:56.322Z      sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(2900352), kind: System }, version: Some(RouterVersion { router_id: 709feadd-7011-4cda-ae1b-097332f1c17e, version: 5 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(df64c47b-92c3-4c28-937e-5a63d4e1ebd5)) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd71:7bbe:8b74::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd71:7bbe:8b74::, width: 64 })) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(df64c47b-92c3-4c28-937e-5a63d4e1ebd5)) }} }
8416  2026-01-27T00:35:56.322Z      sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(2900352), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8417  2026-01-27T00:35:56.322Z      test integration_tests::multicast::failures::test_multicast_group_dpd_communication_failure_recovery has been running for over 60 seconds
8418  2026-01-27T00:35:56.322Z      sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(2900352), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8419  2026-01-27T00:35:56.322Z      sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(2900352), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
8420  2026-01-27T00:35:56.322Z      test integration_tests::multicast::failures::test_multicast_group_dpd_communication_failure_recovery ... FAILED
8421  2026-01-27T00:35:56.322Z
8422  2026-01-27T00:35:56.322Z      failures:
8423  2026-01-27T00:35:56.322Z
8424  2026-01-27T00:35:56.322Z      failures:
8425  2026-01-27T00:35:56.322Z          integration_tests::multicast::failures::test_multicast_group_dpd_communication_failure_recovery
8426  2026-01-27T00:35:56.322Z
8427  2026-01-27T00:35:56.322Z      test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 624 filtered out; finished in 80.34s
8428  2026-01-27T00:35:56.322Z
8429  2026-01-27T00:35:56.322Z    stderr ───
8430  2026-01-27T00:35:56.322Z      log file: /var/tmp/omicron_tmp/test_all-d3fa3cd4a310065e-test_multicast_group_dpd_communication_failure_recovery.2001.0.log
8431  2026-01-27T00:35:56.322Z      note: configured to log to "/var/tmp/omicron_tmp/test_all-d3fa3cd4a310065e-test_multicast_group_dpd_communication_failure_recovery.2001.0.log"
8432  2026-01-27T00:35:56.322Z      DB URL: postgresql://root@[::1]:57189/omicron?sslmode=disable
8433  2026-01-27T00:35:56.322Z      DB address: [::1]:57189
8434  2026-01-27T00:35:56.322Z      log file: /var/tmp/omicron_tmp/test_all-d3fa3cd4a310065e-test_multicast_group_dpd_communication_failure_recovery.2001.2.log
8435  2026-01-27T00:35:56.322Z      note: configured to log to "/var/tmp/omicron_tmp/test_all-d3fa3cd4a310065e-test_multicast_group_dpd_communication_failure_recovery.2001.2.log"
8436  2026-01-27T00:35:56.322Z
8437  2026-01-27T00:35:56.322Z      thread 'integration_tests::multicast::failures::test_multicast_group_dpd_communication_failure_recovery' (2) panicked at sled-agent/src/sim/collection.rs:274:65:
8438  2026-01-27T00:35:56.322Z      called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:44304/vmms/a3384325-1159-454e-b72b-79e247e6da83)" }
8439  2026-01-27T00:35:56.322Z
...

the stack for that unwrap starts at https://buildomat.eng.oxide.computer/wg/0/details/01KFY6CT3MJF8JM8EN06V1YM9P/cl4dgMztjlUA5dUu6PW7VyiY9NUyD82axBUG9IGtLURWEj8f/01KFY6D5X9DJXSA0V9RCPYEKTA#S8439 but I'm not sure which part is relevant here and there is.. a lot of stack.

[zeeshanlakhani]

I had an update for this waiting on a PR that just hasn't received reviews with everything else in play, as we do intentionally fail DPD for some multicast integration tests and need to more flexibly handle cleanup. I'll make a PR of just that change.