Fix ASYNC240 rule findings

[jscheffl]

Apache Airflow version

main (development)

If "Other Airflow 3 version" selected, which one?

No response

What happened?

PR #61417 with upgrading Ruff version to 0.15.0 enabled rule ASYNC240 per default. Some occurrences and findings make the code not compatible.

Nevertheless the rule seems to make sense in general not blocking async code by IO requests.

What you think should happen instead?

Rule should be re-enabled and code locations should be optimized.

NOTE: It is to be decided which of the two libraries to use and it seems this is adding a new dependency. So a short cal to maintainers is needed to get a quorum which to pick.

How to reproduce

Remove exclusion of rule ASYNC240 as it was added by e3c727c in pyproject.toml, run ruff in the repo to see cases where code needs to be fixed.
Follow descriptions in https://docs.astral.sh/ruff/rules/blocking-path-method-in-async-function/#blocking-path-method-in-async-function-async240

Operating System

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[js0547]

@jscheffl I’d like to pick this up as a first contribution if that’s okay.

[sawantbandita]

Hi! I’m a beginner contributor and very interested in working on this issue as well. I’m happy to collaborate or take on part of the ASYNC240 fixes (e.g. identifying affected files or preparing patches/tests). Please let me know how you’d like to proceed.

[v4xsh]

Hi @jscheffl I’d like to work on this. I’ve looked into the ASYNC240 rule and the discussion regarding the dependency quorum for Airflow 3.

The issue is that pathlib methods like .exists() and .is_file() are synchronous and block the event loop when called inside async functions. To re-enable the rule, we need to offload these calls to a thread or use an async filesystem wrapper.

Regarding the library choice mentioned in the description:

• AnyIO: We could use anyio.to_thread.run_sync to wrap the pathlib calls. This is a surgical fix that doesn't require a full filesystem refactor.
• Trio/Aiofiles: These provide native async path objects but would add a more significant dependency for the core.

I’m happy to implement whichever approach the maintainers prefer to get this rule re-enabled.

[jscheffl]

Thanks for the analysis. I am not a deep enought asyncio expert. Any opinions from other experts? @potiuk @ashb @dstandish @dabla ??

[v4xsh]

Thanks for the feedback!

Agreed — this probably needs a maintainer call before locking in an approach. From my side, I don’t have a strong preference yet and would follow whatever direction the core team recommends.

That said, if a low-risk default is helpful for discussion: anyio.to_thread.run_sync feels like the smallest surface-area change to unblock ASYNC240 without introducing a new async filesystem abstraction. But I’m very happy to wait for guidance from @potiuk, @ashb, @dstandish, or @dabla before moving forward.

Let me know how you’d like to proceed.

[jscheffl]

I now had time to look to trio and anyio. I think both are valid but anyio seems to be a bit stronger maintained. If nobody has another opinion in the next 24 hours I'd propose to go with anyio. Anyio has less dependencies.

Other opinions welcome!