[Standard Update]: Allow for links in DATA to a temporary COM space on DATAROOT

[CatherineThomas-NOAA]

What are the limitations of the current standard?

Since GFSv16, the global-workflow has been moving from linking within the DATA directory to copying all of the needed files to and from COM (see bugzilla#1301, NOAA-EMC/global-workflow#712 for examples). This has caused many jobs related to the ensembles and data assimilation to run very long due to the time it takes to copy 80 ensemble members worth of files.

What changes are being proposed to the current standard?

A temporary COM space within the DATAROOT disk is being proposed. We could stage the ensemble member files for a particular cycle there and then each individual job would link to that temporary space within its DATA directory. This would allow the links to be confined to a single disk rather than across disks from DATAROOT to COM while also considerably speeding up job time.

How do these changes improve the current standard?

It allows for greater flexibility in the workflows and more efficiency in several jobs.

What are potential impacts by changing this standard?

Systems with high data volume would benefit from this change, including GFS and RRFS.

How will this standard be enforced?

As part of the SPA implementation code review after code handoff

Additional context or notes (optional)

cc @StevenEarle-NCO

@JacobCarley-NOAA @DavidHuber-NOAA @JessicaMeixner-NOAA @RuiyuSun - Please add details that I may have missed

[JacobCarley-NOAA]

Tagging @aerorahul as well.

I think this request is sensible since it avoids linking across filesystems while also speeding up job execution. My understanding is that we need to decide on this one somewhat quickly so we have the time to update the GFS workflow accordingly.

[StevenEarle-NCO]

Can you give more details about what this looks like: "A temporary COM space within the DATAROOT disk is being proposed."?
Can you also provide information about the data that would be in this temporary COM space? Does it eventually need to be in COM? Is it used by downstream processes within the GFS, is it used between cycles of the GFS, is it used by downstream application of the GFS? There are a couple priorities for production that we need to make sure are maintain:
-- Separation of IO... this is the reason links between filesystems don't work. We need to be able to manage the infrastructure with self contained IO;
-- Ability to rerun / reproduce a failure. Input data to a process, log files, and the ability to save this data without impacting other processes.
-- no public products out of DATA; we need traceability of public products so they need to be retained in COM

[CatherineThomas-NOAA]

Thank you for your additional questions @StevenEarle-NCO. Here are some more details that addresses at least a few of your questions.

These are files that would need to also be in COM, but they would be used linked for use internally in GFS only (edited for clarity), not for downstream applications. They would be used between cycles of the GFS but should only be for neighboring cycles (00 and 06, 06 and 12, etc).

To give an example, this is how it works now for the analysis jobs:

• 00z enkfgdas runs ensemble forecasts and produces hourly history file output (gdas.t00z.atmfFFF.nc): 7 time levels x 80 members. Copy these files from DATA to COM once forecast is complete.
• 06z gfs analysis copies those 7 x 80 forecast files from COM to its DATA directory. Run gfs analysis and read files. Delete those files once gfs_anal is complete.
• 06z gdas analysis copies the same 7 x 80 forecast files from COM to its DATA directory. Run gdas analysis and read files. Delete those files once gdas_anal is complete.
• 06z enkfgdas analysis (eupd) copies the same 7 x 80 forecast files from COM to its DATA directory. This is run the same time as gdas_anal and now we have two copies of same files on DATAROOT concurrently. Run eupd. Delete those files once eupd is completed.

This could be replaced by a space within the DATA disk where we stage those 7x80 files and then when we run the analyses, rather than copying those files from COM to DATA in those 3 jobs, we link to that temporary space instead.
There are some other GFS jobs that would greatly benefit from a structure like this as well.

@aerorahul @DavidHuber-NOAA @JacobCarley-NOAA - Please correct or add to what I've said here.

[CatherineThomas-NOAA]

Also tagging @danholdaway @DavidNew-NOAA for awareness

[aerorahul]

See proposal no. 4 from PersistentDATA
Other suggestions welcome.

[StevenEarle-NCO]

What you described sounds absolutely perfect for production!
....but I completely understand the redundancy and the strain that will put on the filesystem, in addition to the time delay that it could cause.
Rahul shared a presentation where we have been iterating on things like this. I think proposal no. 4 is good, but this was all a paper exercise so it's going to be good to work through it with a real application / workflow like the GFS.

Let's get your initial thoughts on the proposal and feasibility of something like it and then we can work through some details behind it.

Also, I'm not 100% sure, but I'm pretty sure atmf files are used from gdas in downstream applications; we'll need to investigate this a bit more because timeliness into COM will be very important if there are downstream external dependencies. In addition, the atmf files are sent to nomads, so we also have to be careful with that timing: https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20250926/06/atmos/

[CatherineThomas-NOAA]

Thanks @StevenEarle-NCO. Sounds like the next step is for us (GFS project leads, MDC EIB/DA) to work out the details of directory structures, file lists, jobs impacted, timings, etc, and then circle back with you. Does that sounds good to everyone?

[JacobCarley-NOAA]

This sounds good to me.

[StevenEarle-NCO]

Good with me, but I also want to add that we're working through something very similar with RRFS right now. They have enormous amounts of data that need to move between COM and DATA. Whatever we establish for RRFS should also apply to the GFS so we can maintain consistency across our operations.

[JacobCarley-NOAA]

@StevenEarle-NCO Thanks for raising the awareness that this is also being worked for RRFS. I'd like for the RRFS devs to also be a part of the discussion. I'm tagging @MatthewPyle-NOAA and @lgannoaa

[lgannoaa]

This document outlines a proposal to enhance our data management strategy. The goal is to reduce redundant file copies and improve I/O efficiency by making a minor modification to the existing "Umbrella DATA" standard.

1. Background - The Current Standard

Our current architecture, which has been successful in AQM and RRFS, uses two main locations for shared data:

COM Directory: For files that need to be shared across different cycles.

Umbrella DATA Directory: A temporary location within $DATAROOT for files shared between different jobs within the same cycle.

Under the current NCO guidance, jobs from a new cycle must copy (cpreq) any required data from the previous cycle's COM directory.

2. The Challenge & Opportunity

This standard, while effective, creates copy I/O redundancy. For example, a job in a new cycle often only needs to read forecast products from the previous cycle, but it must perform a full copy from COM, which is inefficient.

3. Proposed Solution

I propose we make two small adjustments to the standard:

1. Modify the Umbrella DATA directory naming convention to include the cycle date ($CDATE), preventing it from being overwritten.

    Example: ${DATAROOT}/${model}_umbrella_data/${CDATE}/...

2. Update the NCO standard to formally allow jobs to have read-only access to the date-stamped Umbrella DATA directories from previous cycles.

4. Key Advantages

This approach offers significant benefits:

Eliminates Redundant Copies: Jobs from other cycles can now use symbolic links (ln -s) to read data directly from a previous cycle's Umbrella DATA directory instead of copying it from COM.

Improves Efficiency: Reduces disk I/O and can speed up job startup times.

Maintains Core Standards: The final, official forecast products and RESTART files will still be copied to COMOUT as required. This proposal does not change the final archiving process.

I believe this is a straightforward change that will make our workflows more efficient. I am happy to discuss this further.

Thank you for your consideration.

[StevenEarle-NCO]

A few people from the management/leadership teams talked about this. @aerorahul is going to open a new issue that references this to talk more broadly about the operational standard for sharing data. NCO will have operational requirements early next week, EMC will have development requirements. Then the NCO and EMC technical teams will design a solution, creating a new standard in the process. This will be used by the RRFS, GFS and other massive data systems in the production suite to ensure continued consistency.