Skip to content

Conversation

@gwauge
Copy link
Contributor

@gwauge gwauge commented Jan 21, 2026

It would be great, if the error/retrying logic was customizable.

Specifically, we need to retry on more than just the specified error types, for example OSError.
Additionally, on ClientErrors s3fs currently only retries if one of a few specific patterns matches. Being able to modify this behavior helps with handling non-standard error messages from object storage providers, such as retrying on "please throttle!".

This PR adds global methods add_retryable_error & set_custom_error_handler, which allow users to customize this behavior.

@martindurant
Copy link
Member

Thanks, this is a good idea!
There is quite a lot here, so it will take me some time to review.

@martindurant
Copy link
Member

Everything looks good herem, I am happy to merge as-is, and I would like to get it into the release that I will likely do later today.

One thing you might want to change: it isn't clear in the docs (to me) that the custom error handler applied a condition in addition to the standard ones that were there before: so if your exception matches something retriable, there's no way with this mechanism to not retry it. I can imagine a follow-up to this that gives full flexibility.

@martindurant martindurant merged commit 3bd93fd into fsspec:main Feb 5, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants