Attempt to load CORDEX data

[hot007]

TL;DR: I have been trying to get a (any!) diagnostic to run with CORDEX data but I'm struggling with CMOR the ESMValCore part.

I modified config-developer.yml to define the CORDEX DRS although it's actually the same as BADC's in NCI's case, and in config-user.yml I set rootpath
CORDEX: [/g/data/rr3/publications/CORDEX/output, /g/data/al33/replicas/cordex/output]
and drs set to CORDEX: NCI

I found a recipe that wasn't doing anything too fancy (though the plotting step may fail as it was originally doing whole Earth plots), CMIP7-Hackathon/recipes/atmosphere/recipe_clouds_bias_precip.yml (renamed so as not to break the original).
I modified the datasets section as follows:

  - {driver: CSIRO-BOM-ACCESS1-0, dataset: CSIRO-CCAM-1704, activity: CORDEX, 
      domain: AUS-44i,  institute: CSIRO, project: CORDEX, rcm_version: v1,
      exp: historical, ensemble: r1i1p1, start_year: 1986, end_year: 2005}    
  - {driver: MIROC-MIROC5, dataset: CSIRO-CCAM-1704, activity: CORDEX, domain: AUS-44i,  
      institute: CSIRO, project: CORDEX, rcm_version: v1,
      exp: historical, ensemble: r1i1p1, start_year: 1986, end_year: 2005} 
  - {driver: NOAA-GFDL-GFDL-ESM2M, dataset: CSIRO-CCAM-1704, activity: CORDEX, 
      domain: AUS-44i,  institute: CSIRO, project: CORDEX, rcm_version: v1,
      exp: historical, ensemble: r1i1p1, start_year: 1986, end_year: 2005} 

I found that I have to define dataset and project to have the yamale validator happy, and with the above configuration, ESMValTool was able to find data:

2024-03-13 05:01:18,197 UTC [1907622] DEBUG   esmvalcore._recipe.to_datasets:343 Populating list of datasets for variable pr in diagnostic clouds_bias_pr
2024-03-13 05:01:18,198 UTC [1907622] DEBUG   esmvalcore._recipe.to_datasets:367 Found Dataset: pr, Amon, CORDEX, CSIRO-CCAM-1704, CSIRO-BOM-ACCESS1-0, AUS-44i, CORDEX, historical, r1i1p1
2024-03-13 05:01:18,198 UTC [1907622] DEBUG   esmvalcore._recipe.to_datasets:367 Found Dataset: pr, Amon, CORDEX, CSIRO-CCAM-1704, MIROC-MIROC5, AUS-44i, CORDEX, historical, r1i1p1
2024-03-13 05:01:18,198 UTC [1907622] DEBUG   esmvalcore._recipe.to_datasets:367 Found Dataset: pr, Amon, CORDEX, CSIRO-CCAM-1704, NOAA-GFDL-GFDL-ESM2M, AUS-44i, CORDEX, historical, r1i1p1
2024-03-13 05:01:18,198 UTC [1907622] DEBUG   esmvalcore._recipe.to_datasets:367 Found Dataset: pr, Amon, obs4MIPs, GPCP-V2.2

However, it doesn't understand the CMOR table it's trying to call:
ERROR esmvalcore._recipe.recipe:729 Unable to load CMOR table (project) 'CORDEX' for variable 'pr' with mip 'Amon'
Indeed, CORDEX does not have an Amon table, only mon, but the table is not defined in the datasets list but in diagnostics, so if I change it there it'll fail for the reference Obs data.

If anyone has any idea how to proceed, or better yet can manage to get this or another diagnostic going with CORDEX data, that would be amazing!

Thank you!
Claire and @chuaxr

[hot007]

Update: tried changing the mip in diagnostics to mon, and added mip: Amon to the additional datasets arguments. Got further, looks like it's erroring more regularly now.

[flicj191]

I've had a try and getting an error with fix_metadata
ERROR esmvalcore.preprocessor:348 Failed to run preprocessor function 'fix_metadata' on the data

File "/g/data/xp65/public/apps/med_conda/envs/esmvaltool-0.3/lib/python3.11/site-packages/esmvalcore/cmor/_fixes/cordex/cordex_fixes.py", line 202, in fix_metadata
    f"Coordinate system {coord_system.grid_mapping_name} "
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'grid_mapping_name'

I can come back and look into this though maybe in a couple weeks.

[hot007]

Thanks @flicj191 - did you have to run on an ARE node to get rid of the network errors?
My guess is that this is related to it being a regional model when the diagnostic was originally making global plots... No rush, anything you can figure out is much appreciated!

[hot007]

That error could also indicate a metadata field expected by the CMORiser that isn't in our CORDEX output, obviously it's too late to add now but if that's the case then a) a bespoke CMORiser might be in order and b) we should make sure that field is included in new data being produced for CMIP6-era CORDEX.

[rbeucher]

So CORDEX is not officially supported. We don't have people working on this right now but people are discussing CORDEX support in the community. There are some issues related to IRIS.