-
-
Notifications
You must be signed in to change notification settings - Fork 152
Open
Description
Description
When deploying to a service with more than 100 tasks, ecs deploy will always timeout because the is_deployed() method doesn't paginate the list_tasks API call.
Root Cause
The AWS list_tasks API returns a maximum of 100 task ARNs per call and provides a nextToken for pagination. The current implementation in ecs.py doesn't handle this:
def list_tasks(self, cluster_name, service_name):
return self.boto.list_tasks(
cluster=cluster_name,
serviceName=service_name
)In is_deployed(), this causes the running task count to be capped at 100:
running_tasks = self._client.list_tasks(...) # Returns max 100 tasks
# ...
running_count = self.get_running_tasks_count(...) # Counts max 100
return service.desired_count == running_count # 120 != 100, always FalseSteps to Reproduce
- Have an ECS service with >100 desired tasks (e.g., 120)
- Run
ecs deploy <cluster> <service> --tag <tag> - Observe that deployment times out even though ECS shows deployment completed successfully
Evidence
- ECS
describe-servicesshows deployment completed in ~3 minutes withrolloutState: COMPLETED ecs deploytimes out after the full timeout periodaws ecs list_taskswithout pagination returns only 100 tasks withnextTokenpresent:Tasks returned: 100 nextToken present: True
Suggested Fix
Paginate list_tasks to retrieve all task ARNs:
def list_tasks(self, cluster_name, service_name):
task_arns = []
paginator = self.boto.get_paginator('list_tasks')
for page in paginator.paginate(cluster=cluster_name, serviceName=service_name):
task_arns.extend(page.get('taskArns', []))
return {'taskArns': task_arns}Workaround
Use --timeout -1 (fire-and-forget) and follow up with aws ecs wait services-stable:
ecs deploy <cluster> <service> --tag <tag> --timeout -1
aws ecs wait services-stable --cluster <cluster> --services <service>Environment
- ecs-deploy version: 1.14.0 (also verified bug exists on develop branch)
- Service desired count: 120 tasks
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels