templated creation

[mdsumner]

ok on @rabernat encouragement I'm just going to ask rather than do my homework for some burning questions:

I have 14000 simple netcdf files that are a regular grid with typical 1D lon,lat arrays and a singleton time slice that is predictable with a file per time.

open_dataset gives a perfectly good xarray but rather than open them all with mf, I think I should be be able to open one, set the time to be length 14000 with regular step and inject all the file names. This is probably not ideal but is of interest to me while learning my way around.

I think I saw this mentioned somewhere, I'm super new to python but not to programming or netcdf 🙏

[rabernat]

Thanks for asking this question Michael.

The feature you want is not directly supported by Xarray per se. However, it is possible via the Kerchunk standard and associated python package.

Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF/HDF5, GRIB2, TIFF, …), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly in-situ access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.

Basically, you create a virtual Zarr dataset that maps each data chunk from the underlying files to a Zarr chunk name. You can then open that in Xarray.

Here's a recent blog post that demonstrates what this enables: https://medium.com/pangeo/accessing-netcdf-and-grib-file-collections-as-cloud-native-virtual-datasets-using-kerchunk-625a2d0a9191

[mdsumner]

ok awesome, that looks like my virtual xarray! I've got as far as the output JSON, time for homework for me to get the details.

[rabernat]

Wow that was fast! 🚀