feat(tidy3d): FXC-5311-enable-cached-loading-from-batch-data

[marcorudolphflex]

What changed

• Added in-memory caching for BatchData access when results are already downloaded and the total size of all batch data files stays under a configurable threshold.
• Introduced config.batch_data_cache.max_total_size_gb (with enabled) to control this behavior.
• Updated docs and changelog to document the new configuration.
• Added tests for cache-hit and cache-miss behavior based on total batch size.

Why

• Avoids repeated web.load calls when users iterate or index batch results multiple times, while preventing large batches from staying in memory.

---

Note

Medium Risk
Changes data-loading behavior in BatchData and adds stateful caching, which could impact memory usage and freshness if edge cases aren’t covered; guarded by a size threshold and explicit config flags.

Overview
Speeds up repeated access to td.web.BatchData results by optionally caching loaded task data objects in memory when the batch’s local result files are already present and their combined size is below a configurable threshold.

Introduces a new config section config.batch_data_cache (with enabled and max_total_size_gb) and wires BatchData.load_sim_data() / __getitem__ to reuse cached objects when eligible, while explicitly disabling the behavior for WASM builds and for large/non-downloaded batches. Documentation, changelog, and new unit tests cover cache-hit vs cache-miss behavior based on total file size.

^{Written by Cursor Bugbot for commit 4ad9e26. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[github-actions]

Diff Coverage

Diff: origin/develop...HEAD, staged and unstaged changes

• tidy3d/config/sections.py (100%)
• tidy3d/web/api/container.py (87.5%): Missing lines 644,648-649,651,655

Summary

• Total: 44 lines
• Missing: 5 lines
• Coverage: 88%

tidy3d/web/api/container.py

Lines 640-659

  640             return self._cache_enabled
  641 
  642         self._cache_enabled = False
  643         if WASM_BUILD:
! 644             return False
  645 
  646         try:
  647             cache_config = config.batch_data_cache
! 648         except AttributeError:
! 649             return False
  650         if not cache_config.enabled:
! 651             return False
  652 
  653         max_bytes = int(cache_config.max_total_size_gb * (1024**3))
  654         if max_bytes <= 0:
! 655             return False
  656 
  657         total_size = 0
  658         for task_path in self.task_paths.values():
  659             try:

---

[yaugenst-flex]

Nice!