Skip to content

is_deployed() never completes for services with >100 tasks due to missing pagination in list_tasks #242

@maksle

Description

@maksle

Description

When deploying to a service with more than 100 tasks, ecs deploy will always timeout because the is_deployed() method doesn't paginate the list_tasks API call.

Root Cause

The AWS list_tasks API returns a maximum of 100 task ARNs per call and provides a nextToken for pagination. The current implementation in ecs.py doesn't handle this:

def list_tasks(self, cluster_name, service_name):
    return self.boto.list_tasks(
        cluster=cluster_name,
        serviceName=service_name
    )

In is_deployed(), this causes the running task count to be capped at 100:

running_tasks = self._client.list_tasks(...)  # Returns max 100 tasks
# ...
running_count = self.get_running_tasks_count(...)  # Counts max 100
return service.desired_count == running_count  # 120 != 100, always False

Steps to Reproduce

  1. Have an ECS service with >100 desired tasks (e.g., 120)
  2. Run ecs deploy <cluster> <service> --tag <tag>
  3. Observe that deployment times out even though ECS shows deployment completed successfully

Evidence

  • ECS describe-services shows deployment completed in ~3 minutes with rolloutState: COMPLETED
  • ecs deploy times out after the full timeout period
  • aws ecs list_tasks without pagination returns only 100 tasks with nextToken present:
    Tasks returned: 100
    nextToken present: True
    

Suggested Fix

Paginate list_tasks to retrieve all task ARNs:

def list_tasks(self, cluster_name, service_name):
    task_arns = []
    paginator = self.boto.get_paginator('list_tasks')
    for page in paginator.paginate(cluster=cluster_name, serviceName=service_name):
        task_arns.extend(page.get('taskArns', []))
    return {'taskArns': task_arns}

Workaround

Use --timeout -1 (fire-and-forget) and follow up with aws ecs wait services-stable:

ecs deploy <cluster> <service> --tag <tag> --timeout -1
aws ecs wait services-stable --cluster <cluster> --services <service>

Environment

  • ecs-deploy version: 1.14.0 (also verified bug exists on develop branch)
  • Service desired count: 120 tasks

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions