`cat` fails on flattened `xmap` results due to ClosedInterval time axis

[shy86]

When processing out-of-memory data, I usually split along the time dimension, reduce each chunk with xmap, and then concatenate the results. However, cat(...; dims=:time) fails for xmap results that flatten the time dimension.

The root cause is that xmap returns a time axis of type IntervalSets.ClosedInterval{Date} after reduction, instead of a concrete time value. cat then tries to compare these intervals and fails because isless is not defined for ClosedInterval{Date}.

From a user perspective, a flattened time reduction should correspond to a single representative time (e.g. first, last, or middle of the chunk), not an interval. Returning a concrete time value would make chunk-wise reduction results composable and allow safe concatenation.

It would also be useful if xmap allowed users to explicitly control how the representative time of a reduced axis is chosen (e.g. :first, :last, :middle, or a custom function). Currently, when writing results to file, the midpoint of the interval appears to be used implicitly, which is not always desirable or obvious to the user.

Here is a full simple example:

using YAXArrays
using YAXArrays: YAXArrays as YAX
using Dates

ax1 = (
    YAX.time(Date(2022, 1, 1):Day(1):Date(2022, 1, 3)),
    YAX.lon(1:2),
    YAX.lat(1:2),
)

ax2 = (
    YAX.time(Date(2022, 1, 4):Day(1):Date(2022, 1, 6)),
    YAX.lon(1:2),
    YAX.lat(1:2),
)

y1 = YAXArray(ax1, ones(3, 2, 2))
y2 = YAXArray(ax2, ones(3, 2, 2))

flat_mean(out, x) = (out .= mean(x); nothing)

m1 = xmap(flat_mean, y1 ⊘ :time, output=XOutput())
m2 = xmap(flat_mean, y2 ⊘ :time, output=XOutput())

res = cat(m1, m2; dims=:time)

julia> res = cat(m1, m2; dims=:time)
ERROR: MethodError: no method matching isless(::IntervalSets.ClosedInterval{Date}, ::IntervalSets.ClosedInterval{Date})
The function `isless` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  isless(::Missing, ::Any)
   @ Base missing.jl:87
  isless(::Any, ::Missing)
   @ Base missing.jl:88
  isless(::Char, ::Char)
   @ Base char.jl:223
  ...

julia> m1
┌ 1×2×2 YAXArray{Float64, 3} ┐
├────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────── dims ┐
  ↓ time Sampled{IntervalSets.ClosedInterval{Date}} [2022-01-01 .. 2022-01-03] ForwardOrdered Irregular Intervals{Start},
  → lon Sampled{Int64} 1:2 ForwardOrdered Regular Points,
  ↗ lat Sampled{Int64} 1:2 ForwardOrdered Regular Points
├─────────────────────────────────────────────────────────────────────────────────────────────────────────── loaded lazily ┤
  data size: 32.0 bytes
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[shy86]

I am not sure whether XOutput is intended to allow explicitly specifying the output axis (e.g. XOutput(y1.time[1])). I tried to force the reduced time axis to a single concrete value, but the program unexpectedly hangs:

xmap(flat_mean, y1 ⊘ :time, output = XOutput(y1.time[1]))

[felixcremer]

You can set the dimension in cat like this to circumvent the problem and there you can explicitly define which element of the interval to use.

This is the full call to cat:

julia> res = cat(m1, m2; dims=Ti([y1.time[1], y2.time[1]]))

I like that with the Intervals we keep a lot of the meta information alive which we would loose if we simply chose one value but you are right we would need better control of which value to save to disk or we would have to write proper bounds values.

[shy86]

You can set the dimension in cat like this to circumvent the problem and there you can explicitly define which element of the interval to use.

This is the full call to cat:

julia> res = cat(m1, m2; dims=Ti([y1.time[1], y2.time[1]]))
I like that with the Intervals we keep a lot of the meta information alive which we would loose if we simply chose one value but you are right we would need better control of which value to save to disk or we would have to write proper bounds values.

Thanks for the suggested workaround. While it is effective, it may not be very intuitive for typical users, and documenting or improving this behavior could be helpful. Although Intervals preserve additional metadata, their practical usefulness after flattening reductions may warrant further consideration.