BDD test suite has ~12% failure rate in CI due to flaky lifecycle tests

[teemow]

Summary

The BDD test suite running with --parallel 50 in CI has become flaky, with approximately 12% of CI runs failing (4 failures out of 33 recent runs). Tests pass reliably in isolation locally but fail intermittently under high parallelism.

Affected Scenarios

Scenario	Failures	Failure Pattern
mcpserver-tool-call-lifecycle	2	Timeout at ~31-36s
mcpserver-streamable-http-tool-call-lifecycle	1	Timeout at ~31s
oauth-sso-state-sync-after-login	1	Timeout at ~10.8s

Common Characteristics

1. All failing tests use wait_for_state: 30s for polling state transitions
2. All involve service lifecycle transitions (start/stop/reconnect)
3. Tests pass locally in isolation but fail under high parallelism (50 workers)
4. Timeout pattern: Tests hit exactly 30s+ which matches the wait_for_state timeout

Root Cause Hypotheses

1. Resource contention under high parallelism: 50 parallel muster instances competing for CPU/ports
2. State transition timing: Service state changes may be slower under CI load
3. Mock server startup delays: HTTP/SSE mock servers may take longer to initialize under contention
4. Polling interval too slow: 1s poll interval with 30s timeout = only 30 attempts

Evidence

Recent CI failures:

• https://github.com/giantswarm/muster/actions/runs/21341791094 - mcpserver-streamable-http-tool-call-lifecycle
• https://github.com/giantswarm/muster/actions/runs/21341603315 - oauth-sso-state-sync-after-login
• https://github.com/giantswarm/muster/actions/runs/21341509493 - mcpserver-tool-call-lifecycle
• https://github.com/giantswarm/muster/actions/runs/21340267184 - mcpserver-tool-call-lifecycle

Proposed Investigation

Phase 1: Reliable Reproduction

• Create stress test script to run N iterations locally with 50 parallel workers
• Establish baseline failure rate locally vs CI
• Identify if failures are random or specific to certain scenarios

Phase 2: Instrumentation

• Add timing diagnostics to muster_manager.go (port allocation, process startup, mock server init)
• Add timing diagnostics to test_runner.go (wait_for_state polling, state transitions)
• Collect detailed logs from failed CI runs

Phase 3: Fixes to Consider

• Increase wait_for_state timeout from 30s to 60s for lifecycle tests
• Implement exponential backoff in state polling (start at 500ms, backoff to 5s)
• Add startup jitter to prevent thundering herd in parallel execution
• Consider reducing CI parallelism from 50 to 25

Acceptance Criteria

• Test suite failure rate in CI < 1% over 20+ consecutive runs
• No changes to test behavior or coverage
• Root cause documented for future reference

Related

• Test framework: internal/testing/test_runner.go
• Instance manager: internal/testing/muster_manager.go
• CI workflow: .github/workflows/ci.yaml

[teemow]

Phase 1: Reliable Reproduction - Results

Local Stress Test Results

Ran 20 iterations of muster test --parallel 50 locally:

Metric	Value
Total Runs	20
Passed	20
Failed	0
Failure Rate	0%

Key Findings

1. Tests are stable locally - All 167 scenarios pass consistently with 50 parallel workers when using proper port configuration.

2. Port exhaustion discovered - Running with --parallel 50 at the default --base-port 18000 causes 100% failure due to port exhaustion ("address already in use" errors). The test framework only searches 100 ports (18000-18099), which is insufficient for 50 parallel workers when each instance may require multiple ports (muster + mock servers).

3. Fix: Use higher base port - Using --base-port 30000 eliminates port conflicts and achieves 100% pass rate.

4. CI flakiness is environment-specific - The ~12% failure rate in CI is not reproducible locally, suggesting CI-specific resource constraints (CPU/memory contention on shared runners).

Root Cause Hypothesis (Updated)

The CI failures are likely caused by a combination of:

1. Port exhaustion - CI uses default base port (18000) which may conflict with other services
2. Resource contention - CI runners have limited CPU/memory, causing slower state transitions
3. Network latency - Mock HTTP servers may respond slower under CI load

Recommendations

1. Immediate fix: Update CI workflow to use --base-port 30000 (or higher)
2. Consider reducing parallelism in CI from 50 to 25 workers
3. Increase wait_for_state timeouts from 30s to 60s for lifecycle tests (already done for oauth-sso-state-sync-after-login in commit cb3420f)

Stress Test Script

Created scripts/stress-test.sh for ongoing flakiness monitoring:

./scripts/stress-test.sh [iterations] [parallelism] [base-port]
./scripts/stress-test.sh 20 50 30000  # Run 20 iterations with 50 workers

Next Steps

• Apply --base-port 30000 to CI workflow
• Run 50+ iterations in CI to measure actual failure rate after fix
• If failures persist, proceed to Phase 2 (instrumentation)