Skip to content

fix(util-stream): handle backpressure in ChecksumStream#1869

Merged
kuhe merged 2 commits intosmithy-lang:mainfrom
UnnaturalCabbage:fix/ChecksumStream-backpressure
Feb 23, 2026
Merged

fix(util-stream): handle backpressure in ChecksumStream#1869
kuhe merged 2 commits intosmithy-lang:mainfrom
UnnaturalCabbage:fix/ChecksumStream-backpressure

Conversation

@UnnaturalCabbage
Copy link
Contributor

@UnnaturalCabbage UnnaturalCabbage commented Feb 12, 2026

Description of changes:
Implemented proper backpressure handling in ChecksumStream.

Previously, the _write() method ignored when push() returned false (indicating the internal buffer reached highWaterMark), which led to uncontrolled memory growth when reads were slower than writes.

This change implements the deferred callback pattern: when push() returns false, we store the callback instead of calling it immediately. When downstream requests more data via _read(), we invoke the pending callback to resume upstream flow.

Backstory

I personally faced this issue while downloading large file from S3 to an ECS Fargate local disk (both in us-east-2). And at some point our service ran out of memory. There is also a related bug ticket for AWS SDK v3 opened by another user, but it was unfortunately closed due to an inability to reproduce the issue back then. From what I can tell, this appeared after responseChecksumValidation was set to WHEN_SUPPORTED as the default for aws s3 sdk v3.

How to reproduce:

I created a repository with an example
Screenshot 2026-02-12 at 16 02 24


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@UnnaturalCabbage UnnaturalCabbage requested a review from a team as a code owner February 12, 2026 15:14
@kuhe kuhe self-assigned this Feb 12, 2026
@kuhe kuhe force-pushed the fix/ChecksumStream-backpressure branch from d4ab54f to 0c89953 Compare February 23, 2026 00:48
@kuhe kuhe added the bug This issue is a bug. label Feb 23, 2026
@kuhe kuhe force-pushed the fix/ChecksumStream-backpressure branch from 0c89953 to af10d84 Compare February 23, 2026 01:03
@kuhe kuhe force-pushed the fix/ChecksumStream-backpressure branch from af10d84 to d5dcc1b Compare February 23, 2026 01:18
@kuhe kuhe merged commit ffe1843 into smithy-lang:main Feb 23, 2026
11 checks passed
@kuhe
Copy link
Contributor

kuhe commented Feb 23, 2026

Thank you for your contribution of this fix.

This fix is available in https://www.npmjs.com/package/@smithy/util-stream/v/4.5.13.

This should be brought into your application(s) by running npm up @smithy/util-stream and confirming with npm ls @smithy/util-stream. This should result in a lockfile update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug This issue is a bug.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants