Consider removing `ensure_min_samples` and directly returning fully masked arrays

[aazuspan]

Context

When applying ufuncs to chunked feature arrays with skip_nodata=True, you could easily end up calling the ufunc with an empty array if a chunk contains only NoData. That's a problem for sklearn estimator methods that validate their inputs with ensure_min_samples=1 by default and fail with empty inputs. To avoid that issue, we added a corresponding ensure_min_samples parameter that temporarily fills input arrays with dummy values up to ensure_min_samples before passing them to the ufunc. That works, but the logic is somewhat complex, requires a handful of different validation checks, and is currently constrained to reshaped 2D data.

Proposed change

While ensure_min_samples supports arbitrary minimum samples, the only practical usage that I'm aware of is ensure_min_samples=1 (i.e. don't pass empty arrays), and that case could be handled with a simpler, faster approach of skipping the ufunc call entirely and just returning a fully masked output array. To do that, we would need to use output_sizes, output_dims, output_dtypes, and nodata_output to construct a correctly shaped and filled output (currently we use the partial result of the ufunc call to construct the output array), but I think that should be feasible.

Since the current solution works, this is probably only worth pursuing if there's a meaningful performance improvement and/or the current 2D dimensionality constraint becomes a limitation elsewhere, and if we can't think of other cases where a ufunc would require >1 sample (although you could always set skip_nodata=False as a last resort).

@grovduck, let me know if you see any limitations I'm not thinking of if we were to ditch ensure_min_samples in favor of this approach. This should probably be pretty low priority since the current solution works, but I wanted to record it as a possibility while I was thinking about it.

[grovduck]

@aazuspan, I did a quick search for places in sklearn that use (e.g.) ensure_min_samples=2. It looks like it comes up in some clustering and covariance algorithm, i.e. places where differentiation between samples is important. But that seems like it would mostly come up in estimator/transformer fitting, which is different than the context that we're using it, which is essentially to apply predictions/transformations over features (i.e. ufuncs). So, like you, I can't think of situations where we would need to differentiate 1 versus >1 valid sample.

Is this mostly to simplify the logic or would you expect a meaningful performance improvement as well?

[aazuspan]

Thanks for giving this some thought! I did a similar search and came to the same conclusion.

Is this mostly to simplify the logic or would you expect a meaningful performance improvement as well?

Yeah, that's a good question. It could be a meaningful performance difference under the right circumstances, i.e. if you had a huge number of chunks¹ and could skip the corresponding function calls, or if the ufunc had a high per-call overhead, but I'm not convinced that it would ever be noticeable in practice.

Going back to my original proposal knowing that performance probably shouldn't motivate the change, I think I lean towards shelving this unless the dimensionality becomes a bottleneck elsewhere, but let me know if you have a preference either way.

Footnotes

1. I did a contrived example running kNN prediction over a fully masked 4096x4096 array with 64x64 chunks, and bypassing the ensure_min_samples logic (and the resulting 4096 predict calls) reduced the total time from 6.1s to 3.3s. With more reasonable chunking though, the difference becomes negligible. ↩

[grovduck]

I think I lean towards shelving this unless the dimensionality becomes a bottleneck elsewhere

Shelving seems like an appropriate decision for now. I do have some pretty irregular shaped model regions that would likely include many blocks of masked data (when projected to EPSG:5070), so we could test this as a use-case if we want to revisit at some point. (Yellow line is current modeling bounds.)