Cannot open pango-cmip6 datastore with version 2025.7.9

[rwspicer]

Description

I'm trying to open the esm catalog for pange-cmip6 with the code from the docstring for the function intake.open_esm_datastore. This is failing with the most recent version of intake-esm 2025.7.9. If I install intake-esm 2025.2.3 the code works.

I'm running this on MacOs 14.5 with python 3.13.5. which throws an OSError. I am able to run the example from the readme with on this system.

I've also tried running this on Ubuntu 2022.04.5 with python 3.13., but get a core dump, including when I try to import intake_esm to run show_versions. These issues also are not present with 2025.2.3

What I Did

import intake
url = 'https://storage.googleapis.com/cmip6/pangeo-cmip6.json'
cat = intake.open_esm_datastore(url)

Version information: output of intake_esm.show_versions()

For MacOs:

INSTALLED VERSIONS
------------------

cftime: 1.6.4
dask: 2025.7.0
fastprogress: 1.0.3
fsspec: 2025.7.0
gcsfs: None
intake: 2.0.8
intake_esm: 2025.7.9
netCDF4: 1.7.2
pandas: 2.3.1
requests: 2.32.4
s3fs: None
xarray: 2025.7.1
zarr: 3.1.0

[charles-turner-1]

Hi,

This looks like it might be related to #697.

Can you try setting the threaded variable to false, ie.:

import intake
url = 'https://storage.googleapis.com/cmip6/pangeo-cmip6.json'
cat = intake.open_esm_datastore(url, threaded=False)

This doesn't appear to be an issue with intake-esm itself but an upstream dependency change. However, the fact that you can revert the issue by installing 2025.2.3 is very useful debugging info.

Could I ask you if you could provide a diff of the two environments - it'll be very helpful to track down this issue & actually solve, rather than just avoid it.

Cheers!

[rwspicer]

Adding threaded=False did not change the result.

Here is the changes that occur when I downgrade to 2025.2.3

  Package        Version  Build            Channel           Size
───────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────

  + asciitree      0.3.3  py_2             conda-forge     Cached
  + fasteners       0.19  pyhd8ed1ab_1     conda-forge     Cached

  Downgrade:
───────────────────────────────────────────────────────────────────

  - intake-esm  2025.7.9  pyhd8ed1ab_0     conda-forge     Cached
  + intake-esm  2025.2.3  pyhd8ed1ab_1     conda-forge     Cached
  - numcodecs     0.16.1  py313h2e7108f_0  conda-forge     Cached
  + numcodecs     0.15.1  py313h2e7108f_0  conda-forge     Cached
  - zarr           3.1.0  pyhe01879c_0     conda-forge     Cached
  + zarr          2.18.7  pyhd8ed1ab_0     conda-forge     Cached

[charles-turner-1]

I'm struggling to recreate this error, even without setting threaded=False.

Please could you provide a script that triggers the error (semi-reliably, I know these segfault/coredump issues are intermittent by their very nature), as well as an environment definition file?

[rwspicer]

Hello,

The only code I've been running is this.

import intake
url = 'https://storage.googleapis.com/cmip6/pangeo-cmip6.json'
cat = intake.open_esm_datastore(url, threaded=False)

I was able to resolve this issue on my MacOS machine by updating Polars from 1.31.0 to 1.32.0.

After discovering this, On my Ubuntu machine I also tried updating Polars from 1.31.0 to 1.32.0. . I still had the core dump occur, but polars now displayied an additional runtime warning.

envs/it2/lib/python3.13/site-packages/polars/_cpu_check.py:258: RuntimeWarning: Missing required CPU features.

The following required CPU features were not detected:
    avx2, fma, bmi1, bmi2, lzcnt, movbe
Continuing to use this version of Polars on this processor will likely result in a crash.
Install the `polars-lts-cpu` package instead of `polars` to run Polars with better compatibility.

Hint: If you are on an Apple ARM machine (e.g. M1) this is likely due to running Python under Rosetta.
It is recommended to install a native version of Python that does not run under Rosetta x86-64 emulation.

If you believe this warning to be a false positive, you can set the `POLARS_SKIP_CPU_CHECK` environment variable to bypass this check.

  warnings.warn(

So, I switched to polars-lts-cpu which resolved the Issue on Ubuntu. (I tried both polars-lts-cpu=0.32.0 and polars-lts-cpu=0.31.0 on Ubuntu and the code worked)

[charles-turner-1]

Great, thanks for all the extra info! I'll take a dive into the cause of the issue when I get the chance, that's certainly not where I expected the issues to be coming from.