-
Notifications
You must be signed in to change notification settings - Fork 54
Description
Problem
The email connector's health status remains UP even when experiencing connection issues with the email server. This creates a misleading health status that doesn't accurately reflect the connector's operational state.
Background
During an incident investigation (related to #inc-email-connector-is-not-working-properly), it was discovered that when the email connector encounters connection problems, it continues to show as healthy while retrying indefinitely in the background.
Root Cause Analysis
Based on the discussion in this thread:
- Infinite Retries: The connector is configured with infinite retries using .withBackoff(Duration.of(5, ChronoUnit.SECONDS), Duration.of(1, ChronoUnit.HOURS))
- No Health Status Update: Connection failures during retries don't trigger a health status change to DOWN
- Initial Connection May Succeed: The connector might successfully connect initially but then fail on reconnection attempts, continuing to retry without surfacing the issue via health checks
Expected Behavior
The health status should be set to DOWN when the connector is unable to establish or maintain a connection to the email server, especially during prolonged retry scenarios.
Proposed Solution
Update the health check logic in the JakartaEmailListener to properly reflect connection failures. Specifically, consider updating the health status in this method when connection issues are detected.
Additional Context
- The
whenCompleteblock following the connection attempt should handle exceptions and update health accordingly - Currently, only the "could not reconnect to store" exception triggers a health down status
- The connector may be experiencing issues with slow or unstable email servers that intermittently fail