Best ways to transfer small-chunked zarr datasets

[timonmerk]

I need to transfer a zarr dataset, and the transfer throughput is unfortunately prohibitively slow. This is likely due to the many files resulting from smaller chunk size that is optimal in our downstream application.
Currently I am using tar to compress individual blocks that makes the transfer faster. I am not sure though if this is the optimal solution, since it requires additional manual steps.
Unfortunately rechunking might break a lot of our chunk-associated metadata.

[d-v-b]

hi @timonmerk, there are a lot of ways to copy zarr data; where are you transferring the data from, and where are you transferring it to? Are you sure the slowness has anything to do with zarr itself, and not your network connection?

[timonmerk]

Thanks for the quick reply @d-v-b. I am transferring from one HPC to another using Globus. I know that the overall throughput time on different (non-zarr) datasets is faster. That's why I could rule out that the network connection is overall slow.
Would you except a slow-down due to the many files?

[d-v-b]

yes, it's likely that you are suffering from per-file overhead. Resaving the data using shards would cost you some compute but might make the transfer much faster.

[timonmerk]

Thanks! Sharding definitely helped the transfer speed a lot! I think one bottleneck is still the many zarr.json files incorporated in the file structure (based on attrs per array), which is probably not preventable

[d-v-b]

does the consumer need the entire thing, or just part of it? hosting the dataset over http from HPC 1, and having the consumer on HPC 2 access (and potentially cache) just the parts they need could be an alternative to globus.

[d-v-b]

see https://github.com/ekzhang/bore and related tools