Support warm-up of loader caches by jakob-keller · Pull Request #1462 · aio-libs/aiobotocore

jakob-keller · 2026-01-02T21:00:13Z

Description of Change

Support warm-up of loader caches

Assumptions

Replace this text with any assumptions made (if any)

Checklist for All Submissions

Checklist when updating botocore and/or aiohttp versions

I have read and followed CONTRIBUTING.rst
I have updated test_patches.py where/if appropriate (also check if no changes necessary)
I have added URL to diff: boto/botocore@[CURRENT_BOTO_VERSION_TAG]...[NEW_BOTO_VERSION_TAG]

gemini-code-assist · 2026-01-02T21:00:16Z

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

codecov · 2026-01-02T21:07:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.67%. Comparing base (3068542) to head (bbc0134).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1462      +/-   ##
==========================================
+ Coverage   91.60%   91.67%   +0.07%     
==========================================
  Files          76       76              
  Lines        8078     8148      +70     
==========================================
+ Hits         7400     7470      +70     
  Misses        678      678

Flag	Coverage Δ
no-httpx	`88.56% <100.00%> (+0.09%)`	⬆️
os-ubuntu-24.04	`91.67% <100.00%> (+0.07%)`	⬆️
os-ubuntu-24.04-arm	`89.70% <100.00%> (+0.08%)`	⬆️
python-3.10	`89.66% <100.00%> (+0.08%)`	⬆️
python-3.11	`89.66% <100.00%> (+0.08%)`	⬆️
python-3.12	`89.66% <100.00%> (+0.08%)`	⬆️
python-3.13	`89.66% <100.00%> (+0.08%)`	⬆️
python-3.14	`91.63% <100.00%> (+0.07%)`	⬆️
python-3.9	`89.67% <100.00%> (+0.08%)`	⬆️
unittests	`91.67% <100.00%> (+0.07%)`	⬆️
with-awscrt	`91.30% <100.00%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jakob-keller · 2026-01-02T21:08:34Z

aiobotocore/session.py

        )
        return service_data

+    def warm_up_loader_caches(


For advanced use cases, including non-default executors, this method may be called explicitly before client creation.

webknjaz · 2026-01-18T21:21:24Z

aiobotocore/session.py

+    def warm_up_loader_caches(
+        self,
+        service_name: str,
+        api_version: Optional[str] = None,


(nit) optional is considered confusing. Best use union of None and whatever's "optional".

We still support Python 3.9 and cannot use the modern str | None syntax introduced in Python 3.10. Those deprecated Optional and Union annotations will be replaced once botocore and we drop support for Python 3.9.

tests/test_session.py

webknjaz · 2026-01-18T21:26:07Z

I can't really review this in-depth, just left a few infra/testing comments.

jakob-keller · 2026-01-18T21:32:23Z

I can't really review this in-depth, just left a few infra/testing comments.

Fair enough and thank you anyway!

patrickvorgers · 2026-01-20T20:13:36Z

I tried to validate this in Home Assistant (IDrive e2 backup provider), but Home Assistant is currently pinned to aiobotocore 2.x. Would it be possible to make this warm-up API available (or backported) on the 2.x line as well, so downstream projects pinned to 2.x can adopt and test it?

jakob-keller · 2026-01-20T20:22:22Z

Would it be possible to make this warm-up API available (or backported) on the 2.x line as well, so downstream projects pinned to 2.x can adopt and test it?

We typically don't provide backports and there are no plans to publish additional 2.x releases.

It should be straightforward to experiment with the code snippets I posted previously or the proposed changes in this PR. That should work with any version of aiobotocore, including 2.x releases.

patrickvorgers · 2026-01-20T20:24:56Z

Home Assistant is currently using aiobotocore 2.21.1 so it is probably good anyhow to check whether it is possible to upgrade to the latest version.

jakob-keller · 2026-01-20T20:27:45Z

Home Assistant is currently using aiobotocore 2.21.1 so it is probably good anyhow to check whether it is possible to upgrade to the latest version.

Possibly, but that won't change anything: You still need to duplicate my proposed cache warming code, if you want to test it. So far, it's only contained in this pending PR and we are awaiting community feedback before we agree on a way to move forward.

patrickvorgers · 2026-01-20T21:11:35Z

I applied the change manually to aiobotocore 2.21.1 and preload the caches.

session = AioSession()
await hass.async_add_executor_job(session.warm_up_loader_caches, "s3")

Some detections like the listdir are gone now but a SSL one remains:

2026-01-20 21:02:16.696 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to load_verify_locations with args (<ssl.SSLContext object at 0x7f191de3f110>, '/home/vscode/.local/ha-venv/lib/python3.13/site-packages/certifi/cacert.pem', None, None) inside the event loop by integration 'idrive_e2' at homeassistant/components/idrive_e2/__init__.py, line 46: await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET]) (offender: /home/vscode/.local/ha-venv/lib/python3.13/site-packages/aiobotocore/httpsession.py, line 165: ssl_context.load_verify_locations(ca_certs, None, None)), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+idrive_e2%22
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#load_verify_locations
Traceback (most recent call last):
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
    cli.main()
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 508, in main
    run()
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 391, in run_module
    run_module_as_main(options.target, alter_argv=True)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 228, in _run_module_as_main
    return _run_code(code, main_globals, None, "__main__", mod_spec)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
  File "/workspaces/core/homeassistant/__main__.py", line 229, in <module>
    sys.exit(main())
  File "/workspaces/core/homeassistant/__main__.py", line 215, in main
    exit_code = runner.run(runtime_conf)
  File "/workspaces/core/homeassistant/runner.py", line 289, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 712, in run_until_complete
    self.run_forever()
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 683, in run_forever
    self._run_once()
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 2042, in _run_once
    handle._run()
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
  File "/workspaces/core/homeassistant/config_entries.py", line 928, in async_setup_locked
    await self.async_setup(hass, integration=integration)
  File "/workspaces/core/homeassistant/config_entries.py", line 672, in async_setup
    await self.__async_setup_with_context(hass, integration)
  File "/workspaces/core/homeassistant/config_entries.py", line 762, in __async_setup_with_context
    result = await component.async_setup_entry(hass, self)
  File "/workspaces/core/homeassistant/components/idrive_e2/__init__.py", line 46, in async_setup_entry
    await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET])

jakob-keller · 2026-01-20T21:15:27Z

I applied the change manually to aiobotocore 2.21.1 and preload the caches.
session = AioSession()
await hass.async_add_executor_job(session.warm_up_loader_caches, "s3")
Some detections like the listdir are gone now but a SSL one remains:

Sounds to me like a confirmation that the proposed approach works in the HA use case.

patrickvorgers · 2026-01-21T08:26:11Z

I applied the change manually to aiobotocore 2.21.1 and preload the caches.
session = AioSession()
await hass.async_add_executor_job(session.warm_up_loader_caches, "s3")
Some detections like the listdir are gone now but a SSL one remains:
Sounds to me like a confirmation that the proposed approach works in the HA use case.

Is there also a way to get rid of the SSL one?

ssl_context.load_verify_locations(ca_certs, None, None)

jakob-keller · 2026-01-21T08:27:49Z

Is there also a way to get rid of the SSL one?

Glancing at the stacktrace you shared, I don't see how that would be related to aiobotocore.

patrickvorgers · 2026-01-21T08:30:52Z

Is there also a way to get rid of the SSL one?

Glancing at the stacktrace you shared, I don't see how that would be related to aiobotocore.

It seems to be triggered by:

await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET])

Which is part of aiobotocore.

jakob-keller · 2026-01-21T10:39:46Z

Is there also a way to get rid of the SSL one?

Glancing at the stacktrace you shared, I don't see how that would be related to aiobotocore.

It seems to be triggered by:
await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET])
Which is part of aiobotocore.

Unblocking setup of the SSL context is another can of worms. This is out of scope of this PR. Please create a separate issue.

thehesiod · 2026-01-22T14:37:06Z

the SSL context thing sounds like it could be easily handled by my version where we have an executor. sorry haven't had a chance to review the comments across all the issues yet in case this was already discussed, will try to catch up soon

patrickvorgers · 2026-01-22T19:40:45Z

the SSL context thing sounds like it could be easily handled by my version where we have an executor. sorry haven't had a chance to review the comments across all the issues yet in case this was already discussed, will try to catch up soon

It would be great if there is already a solution in place that could work.
Any ideas on how to verify this and move this forward?

jakob-keller added the enhancement New feature or request label Jan 2, 2026

jakob-keller force-pushed the warm-up-loader-caches branch from 726bb38 to eca8210 Compare January 2, 2026 21:03

jakob-keller mentioned this pull request Jan 2, 2026

Add support for executor configuration in AioConfig for file loading #1451

Open

10 tasks

jakob-keller commented Jan 2, 2026

View reviewed changes

jakob-keller mentioned this pull request Jan 2, 2026

Detected blocking call inside the event loop #1199

Open

6 tasks

jakob-keller force-pushed the warm-up-loader-caches branch 2 times, most recently from d2bc4da to 74d9cef Compare January 3, 2026 00:27

jakob-keller added the bug Something isn't working label Jan 3, 2026

jakob-keller force-pushed the warm-up-loader-caches branch 2 times, most recently from 7eb8690 to c60df2e Compare January 3, 2026 03:36

jakob-keller marked this pull request as ready for review January 3, 2026 03:40

jakob-keller requested review from bdraco, thehesiod and webknjaz January 3, 2026 03:42

patrickvorgers mentioned this pull request Jan 18, 2026

IDrive e2 backup provider home-assistant/core#144910

Merged

19 tasks

webknjaz reviewed Jan 18, 2026

View reviewed changes

tests/test_session.py Outdated Show resolved Hide resolved

jakob-keller force-pushed the warm-up-loader-caches branch 3 times, most recently from 53c2115 to 95b54a2 Compare January 20, 2026 16:43

patrickvorgers mentioned this pull request Jan 21, 2026

Unblocking creation of SSL context #1469

Open

6 tasks

jakob-keller force-pushed the warm-up-loader-caches branch from 95b54a2 to 42d671d Compare January 29, 2026 20:49

Support warm-up of loader caches

bbc0134

jakob-keller force-pushed the warm-up-loader-caches branch from 42d671d to bbc0134 Compare February 5, 2026 10:04

Uh oh!

Conversation

jakob-keller commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Change

Assumptions

Checklist for All Submissions

Checklist when updating botocore and/or aiohttp versions

Uh oh!

gemini-code-assist bot commented Jan 2, 2026

Uh oh!

codecov bot commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jakob-keller Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz Jan 18, 2026

Choose a reason for hiding this comment

Uh oh!

jakob-keller Jan 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

webknjaz commented Jan 18, 2026

Uh oh!

jakob-keller commented Jan 18, 2026

Uh oh!

patrickvorgers commented Jan 20, 2026

Uh oh!

jakob-keller commented Jan 20, 2026

Uh oh!

patrickvorgers commented Jan 20, 2026

Uh oh!

jakob-keller commented Jan 20, 2026

Uh oh!

patrickvorgers commented Jan 20, 2026

Uh oh!

jakob-keller commented Jan 20, 2026

Uh oh!

patrickvorgers commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakob-keller commented Jan 21, 2026

Uh oh!

patrickvorgers commented Jan 21, 2026

Uh oh!

jakob-keller commented Jan 21, 2026

Uh oh!

thehesiod commented Jan 22, 2026

Uh oh!

patrickvorgers commented Jan 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jakob-keller commented Jan 2, 2026 •

edited

Loading

codecov bot commented Jan 2, 2026 •

edited

Loading

jakob-keller Jan 2, 2026 •

edited

Loading

patrickvorgers commented Jan 21, 2026 •

edited

Loading