feat(s3): use AWS SDK v2 TransferManager for copy task

[nancysangani]

Description

Implements AWS SDK for Java v2 TransferManager for S3 copy operations to improve performance through automatic parallelization and multipart-aware copying.

Changes

• Replaced synchronous S3Client with S3AsyncClient and S3TransferManager
• Enabled automatic multi-threaded copy operations for better throughput
• Fixed critical bug: KMS key ID is now actually applied when using AWS_KMS encryption (field was defined but never used)
• Improved error handling for delete flag with .orElse(false)

Performance Impact

• Single-threaded → Multi-threaded parallel copy
• Automatic multipart copy for large files (>5GB)
• Significant throughput improvement for batch operations

Implementation Details

The TransferManager automatically handles:

• Parallel uploads for multipart operations
• Optimal part size calculation
• Retry logic for failed parts
• Resource management

Fixes

Closes #661

[Copilot]

Pull request overview

This PR migrates the S3 Copy task from using the synchronous S3Client to the AWS SDK v2 TransferManager with S3AsyncClient, enabling automatic parallelization and multipart-aware copying for improved performance. The PR also fixes a critical bug where the KMS key ID field was defined but never actually applied when using AWS_KMS encryption.

Changes:

• Replaced synchronous S3Client with S3AsyncClient and S3TransferManager for parallel copy operations
• Fixed critical bug: KMS key ID is now properly applied when using AWS_KMS server-side encryption (lines 122-126)
• Improved error handling for the delete flag using .orElse(false) instead of .orElseThrow() (line 143)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[fdelbrayelle]

Hi @nancysangani 👋 Could you provide some screenshots showing how you tested it with AWS and Kestra UI please?
Plus didn't you have to update the unit/integration tests accordingly?

[nancysangani]

Hi @fdelbrayelle, sure, I’ll add the screenshots of the tests and update the tests if needed. Will share them ASAP.

[nancysangani]

Hi @fdelbrayelle, I tested this using a local setup with Kestra + LocalStack (S3) via Docker and verified the behavior through the Kestra UI.

Testing with Kestra UI:

Here is a screen recording showing the flow configuration in the Kestra UI and a successful execution:

Video.Project.-.AWS.mp4

Video.Project.-.AWS.Copy.mp4

Regarding test updates:

No modifications to the test files were required. The migration from S3Client to S3TransferManager is an internal implementation change that doesn't affect the public API, so all existing tests pass without modification.