An Infinite loop during application startup when a queue does not exist

[tomaszmwrobel]

Type: Bug

Component:
SQS

Describe the bug
SC AWS version: 3.4.0
When not-found-strategy is set to FAIL (spring.cloud.aws.sqs.queue-not-found-strategy=FAIL) and an expected queue does not exist, the application falls into an infinite loop with undefined state. The undefined state means that Tomcat server is stopped, the application is not working, but the JVM process is still running, e.g docker container is working and consuming resources until a host sends SIGKILL or SIGTERM signal.

Additionally if an SqsListener is created programatically by SqsMessageListenerContainer the other defined queue start consuming messages despite invalid configuration of one SqsListener. This message processing might be interrupted by graceful shutdown (due to invalid configuration) leading to the potential loss of messages that have already been consumed.

Sample

@SqsListener("${correct.queue.name}") public void processEvent(EventDTO eventDto) { //anything }

@Bean SqsMessageListenerContainer<GreetingEvent> sqsMessageListenerContainer(SqsAsyncClient sqsAsyncClient) { return new SqsMessageListenerContainer.Builder<GreetingEvent() .sqsAsyncClient(sqsAsyncClient).queueNames("another-correct-queue") .messageListener((message) -> { log.info("Received message {}", message); }).build(); }

@SqsListener("invalid-queue-name") public void onInvalidMessage(GreetingEvent event) { }

[brun0-4ugusto]

Hello, @tomaszmwrobel ! @tomazfernandes , could this bug be related to the discussion in this issue #1143 ? We can try to fix it and dig a bit deeper into this behavior. If you’d like, I can send a PR suggestion, if that works for you.

[klach-ocado]

Hi @brun0-4ugusto, I work with @tomaszmwrobel and I'll look at that but next week (integration with v4.0.0-RC1 has higher priority :) ). Thanks.

[tomazfernandes]

Hey @tomaszmwrobel and @klach-ocado,

When not-found-strategy is set to FAIL (spring.cloud.aws.sqs.queue-not-found-strategy=FAIL) and an expected queue does not exist, the application falls into an infinite loop with undefined state. The undefined state means that Tomcat server is stopped, the application is not working, but the JVM process is still running, e.g docker container is working and consuming resources until a host sends SIGKILL or SIGTERM signal.

Can you share full logs from startup to shutdown, ideally with io.awspring.cloud.sqs at DEBUG level, plus your Spring Boot and Java versions and whether this is docker or local?

Additionally if an SqsListener is created programatically by SqsMessageListenerContainer the other defined queue start consuming messages despite invalid configuration of one SqsListener.

Depending on how that programmatic container is created / started (bean-managed vs manual start()), this may be expected. Could you confirm whether it's declared as a Spring Bean and whether Spring is starting it?

This message processing might be interrupted by graceful shutdown (due to invalid configuration) leading to the potential loss of messages that have already been consumed.

With default acknowledgment configuration, the message will only be deleted after processing. If processing is interrupted, the message should be visible again after visibility timeout, which is consistent with the at-least-once model in SQS.

Thanks.

[tomazfernandes]

@brun0-4ugusto thanks for linking #1143, useful context. Let’s confirm from the logs/environment details first, then we can see whether this is hitting the same not-found handling path. If you have any thoughts on where the FAIL path might be looping or not stopping cleanly, feel free to share.

[klach-ocado]

Please see the minimal reproducible example: https://github.com/klach-ocado/sqs-listener-infinite-loop

Reproducible on Spring Boot 3.5.9 + Spring Cloud AWS 3.4.2, as well as on Spring Boot 4.0.1 + Spring Cloud AWS 4.0.0-RC1. To test with the latter, proper versions need to be uncommented in pom.xml and proper SqsMessagingMessageConverter in MainConfiguration.

[klach-ocado]

One more thing I forget: The above example reproduces failure at application startup. We also have an integration where SQS listeners do not start automatically (io.awspring.cloud.sqs.listener.ContainerOptionsBuilder#autoStartup called with false during configuring beans) but instead the application detects whether load balancer switches traffic to the given instance (from another instance from previous deployment), and then starts SQS listeners (containers to be precise) manually. And as that's already after the application started successfully, it's hard to handle that case properly. But if getting a queue's URL was always executed at application startup, even when SQS's auto-startup is false, then the application could fail early, and later only start processing messages, but we'd already know that queues exist. I give that as an idea.

[klach-ocado]

Hi @brun0-4ugusto,

could this bug be related to the discussion in this issue #1143 ?

I'd say there are similarities but they are about different needs. The mentioned issue is about currently having no "ignore" / "no-op" strategy. This issue is about the "fail" strategy currently being unable to stop the application if some queues started properly while some failed. The #1143 requires the application to continue to work even if all queues do not exist (and "ignore" strategy is selected), this issue requires the application to stop working even if some queues started (when "fail" strategy is selected).

Why we need fail fast? We deploy apps to multiple AWS accounts for different customers, and it's possible that some misconfiguration may happen (e.g. CloudFormation or Terraform deployment for provisioning resources fails or is not in sync with application's deployment), and then we'd like to know instantly that there is a problem.

[tomazfernandes]

Hey @klach-ocado, thanks for the sample.

The problem is that in the start() method in DefaultListenerContainerRegistry we set this.running to true only after all containers have started:

spring-cloud-aws/spring-cloud-aws-sqs/src/main/java/io/awspring/cloud/sqs/listener/DefaultListenerContainerRegistry.java

Lines 80 to 89 in eba0d3f

	public void start() {
	synchronized (this.lifecycleMonitor) {
	logger.debug("Starting {}", getClass().getSimpleName());
	List<MessageListenerContainer<?>> containersToStart = this.listenerContainers.values().stream()
	.filter(SmartLifecycle::isAutoStartup).collect(Collectors.toList());
	LifecycleHandler.get().start(containersToStart);
	this.running = true;
	logger.debug("{} started", getClass().getSimpleName());
	}
	}

At the same time, in Spring's DefaultLifecycleProcessor, the stop() method is not called on beans that respond false to isRunning():

https://github.com/spring-projects/spring-framework/blob/7917ae53e80214760e8590e3d1f9c0e91bb14b69/spring-context/src/main/java/org/springframework/context/support/DefaultLifecycleProcessor.java#L468-L508

So, when a container fails to start, Spring's DefaultLifecycleProcessor skips the DefaultListenerContainerRegistry and results in the containers not being shut down.

The fix would be to move this.running=true to the beginning of the start() method in DefaultListenerContainerRegistry, so when a container fails to start the stop method is called appropriately when the application shutsdown.

As a workaround, with this implementation the application correctly shuts down when the container fails:

	@Component
	public static class LifecycleOverrider implements SmartLifecycle, ApplicationContextAware {

		private MessageListenerContainerRegistry registry;

		private boolean running = false;

        @Override
		public void start() {
			this.running = true;
		}

		@Override
		public void stop() {
			this.registry.stop();
			this.running = false;
		}

		@Override
		public int getPhase() {
			return 0;
		}

		@Override
		public boolean isRunning() {
			return this.running;
		}

		@Override
		public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
			this.registry = applicationContext.getBean(MessageListenerContainerRegistry.class);
		}
	}

A PR with the fix would be welcome, ideally with a test that fails under the current implementation and passes after the fix.

[klach-ocado]

Hi @tomazfernandes,

Thank you for the above analysis. And regarding a potential PR with the fix, at the moment we have other priorities but I'll keep this in mind and return to this later.

Thanks.