ManifestStore.to_zarr()

[TomNicholas]

@ilangold asked if there was any pre-existing tool for converting HDF5 files directly into Zarr stores, and it occurred to me that you could do this very efficiently using ManifestStore:

1. Call the Parser on the filepath for hdf5/other file to determine all chunk locations and metadata,
2. Use obstore to get all the chunk data key-by-key,
3. Use obstore to put all the chunk data and metadata into the new store location key-by-key. Voila!

Pros and cons compared to using xarray

Of course you could alternatively use VirtualiZarr like a runtime translation layer and then use xarray's .to_zarr() like this:

from virtualizarr.parsers import TiffParser

manifest_store = TiffParser("<file.tiff>")

ds = xr.open_zarr(manifest_store)

ds.to_zarr('output_path.zarr')

but that has a couple of significant downsides:

• Using zarr-python (here via xarray) to read chunks will decompress them, which is pointless here as you know you're about to just recompress them before writing them again,
• Xarray's open_dataset/zarr does other complicated decoding steps that you also don't need here,
• Xarray writes zarr with some extra conventions, particularly adding a coordinates metadata field,
• Xarray Datasets / DataTrees cannot represent all Zarr stores.

The main downsides of this suggestion are that:

• It would only work for file formats that are virtualizable (i.e. they can be read from efficiently using targeted byte range requests), whereas there are other file formats which are not virtualizable but whose data is still convertable to zarr (e.g. tabular formats).
• You couldn't rechunk on the way, at least not without significantly complicating the implementation.
• The API gives you little opportunity to drop arrays or chunks.

Given those restrictions, I'm not sure this idea really offers anything beyond your existing two options of

1. Creating a virtual icechunk store (either using xarray right now or even without xarray if we merge ManifestStore.to_icechunk() - see Add .to_icechunk() method to ManifestGroup #591).
2. Creating a native zarr store using xarray.

So I'm not sure this is that useful, but it's interesting! 😆