Uniquely identify AWS resources for Dask clusters running in the same ECS cluster

[dicknetherlands]

Addresses #472 by including the unique self.name in all Dask cluster AWS resource names auto-created within the same ECSCluster/FargateCluster, and tagging them with it too.

Previously, cluster_name was used for this purpose, but when running multiple simultaneous Dask clusters on the same ECS cluster (by specifying cluster_arn) while still requiring Dask to auto-generate some AWS resources such as TaskDefinitions, this caused name collisions because cluster_name maps to the ECS cluster name and is not unique. self.name, however, is a UUID by default, so is unique by default unless the user explicitly overrides it.

This patch resolves the collision situation and also makes resources associated with an individual Dask cluster much easier to identify.

The existing code would not support more than one Dask cluster on the same ECS cluster unless all resources were provided. Therefore, anyone already setting the self.name property to anything other than the default random unique value would not be affected by this change. However, in new code, setting self.name to a non-unique value will cancel out the ability to run more than one Dask cluster on the same ECS cluster unless all resources are provided.

[jacobtomlinson]

I think the challenge here is we've assumed a 1:1 mapping between an ECS cluster and the Dask cluster running on it. This is why the names are so entangled.

The cluster_name refers to the ECS cluster, but is used in a bunch of other places too.

Every cluster object also has a name property too, could that not be used for the ID?

[dicknetherlands]

Every cluster object also has a name property too, could that not be used for the ID?

Yes! I didn't notice that. I've updated it to use that instead of introducing a new ID field. It defaults to a UUID already which is exactly what we need.

The PR doesn't interfere with existing uses of cluster_name except for replacing/augmenting its use in AWS resource names to reference self.name to make them unique to this cluster instance, and include both in the tags.

[jacobtomlinson]

This looks great thanks. Can you confirm that tests for ECS are passing locally for you?

[dicknetherlands]

Yes, they pass… but, if you look at the ecs module tests they’re not exactly comprehensive!

…

Sent from my iPhone On 14 Aug 2025, at 9:58 pm, Jacob Tomlinson ***@***.***> wrote:  @jacobtomlinson commented on this pull request. This looks great thanks. Can you confirm that tests for ECS are passing locally for you? — Reply to this email directly, view it on GitHub<#474 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE2HD6OWPBMAZDZJDYRK6Q33NRMR7AVCNFSM6AAAAACDYAXL3SVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTCMJZHEZTGOBYGA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jacobtomlinson]

That's a good point. Could I ask you to add a couple of tests that verify you can start multiple Dask clusters that share the same ECS cluster?

[dicknetherlands]

That's a good point. Could I ask you to add a couple of tests that verify you can start multiple Dask clusters that share the same ECS cluster?

@jacobtomlinson Done.

[jacobtomlinson]

Fantastic many thanks!