Generating references from files in S3 (using kerchunk + fsspec)

[TomNicholas]

Raised by @abarciauskas-bgse in https://github.com/TomNicholas/VirtualiZarr/issues/60.

Presumably kerchunk's backends opener functions can already do this? In which case we just need to pass the right kwargs / fsspec whatevers to those inside open_virtual_dataset.

[norlandrhagen]

Wondering how we want to package up kwargs for all the different kerchunk file readers so we can pipe in things like storage_options to get s3 reading to work.

netcdf3, netcdf4, tiff etc. all have a few options. ex:

SingleHdf5ToZarr:

  h5f: 'BinaryIO | str | h5py.File | h5py.Group',
  url: str = None,
  spec=1,
  inline_threshold=500,
  storage_options=None,
  error='warn',
  vlen_encode='embed',
  out=None,


NetCDF3ToZarr:

def __init__(
    self,
    filename,
    storage_options=None,
    inline_threshold=100,
    max_chunk_size=0,
    out=None,
    **kwargs)

tiff_to_zarr:

    urlpath: str
        Location of input TIFF
    remote_options: dict
        pass these to fsspec when opening urlpath
    target: str
        Write JSON to this location. If not given, no file is output
    target_options: dict
        pass these to fsspec when opening target

etc..

[TomNicholas]

Urgh that's messy - I'm not familiar at all with these so would defer to your judgement. But generally they should all go in one dict so that they can be cleanly separated from Xarray kwargs once open_virtual_dataset is replaced by xr.open_dataset(..., engine='virtualizarr').

[TomNicholas]

Whilst I think the approach in #67 is fine for now, is it possible to imagine any other implementation of reading byte ranges from remote files that doesn't rely on fsspec / kerchunk?

[rabernat]

is it possible to imagine any other implementation of reading byte ranges from remote files that doesn't rely on fsspec / kerchunk?

It's straightforward to abstract this via a wrapper class. Just imagine the API you'd like to use from the virtualizarr perspective, create an ABC interface for it, and then make an fsspec implementation.

[norlandrhagen]

@TomNicholas that would be nice! For a stop-gap, happy to continue working on #67 or if you want to pause and look for other file reader paths sounds good to.

[TomNicholas]

I really have no idea, since I've not worked with fsspec very much at all. Input welcome.

I guess from the virtualizarr perspective I just want to be able to pass in a filepath / S3 url directly to the open_virtual_dataset function, and not have to create a bunch of weird filesystem mapper objects first. I also don't want to have to pass a bunch of specific reader options or whatever (why doesn't that part "just work"...?) I don't think we need to support reading from anything other than posix filesystems and s3 urls anyway. The main thing to avoid here is just copying kerchunk's fsspec arguments into open_virtualizarr.

[abarciauskas-bgse]

@TomNicholas curious about this comment https://github.com/TomNicholas/VirtualiZarr/issues/61#issuecomment-2027723156

is it possible to imagine any other implementation of reading byte ranges from remote files that doesn't rely on fsspec / kerchunk?

since the current implementation relies on reading files using kerchunk - are you thinking about changing that implementation entirely?

If keeping the current implementation, which relies on kerchunk modules, I think #67 makes sense as that PR just builds on that integration via the additional parameters kerchunk needs to open remote files.

[TomNicholas]

since the current implementation relies on reading files using kerchunk - are you thinking about changing that implementation entirely?

Well looking forward, we want to eventually use zarr chunk manifests for storing the byte ranges on disk, which leaves the only part of this library that actually depends on kerchunk being the kerchunk backends for reading the byte ranges. It's at least interesting to imagine what we could implement instead of using kerchunk to read into a format that we have to convert before using anyway.

EDIT: IIUC with zarr v3 and the rust object-store create we also wouldn't need fsspec to actually read data from disk/object storage either. (I wonder if it's possible to use the same crate to read just byte ranges?...)

I'm only floating this as an idea, not a concrete plan though.

[abarciauskas-bgse]

IIUC with zarr v3 and the rust object-store create we also wouldn't need fsspec to actually read data from disk/object storage

I would say it's ok to postpone S3 support until some of those things you mentioned (zarr chunk manifests, zarr v3 rust object-store) exists - wdyt @norlandrhagen

Is there already WIP rust object-store @TomNicholas

[TomNicholas]

The rust crate exists, but you'll have to ask @jhamman about the status of attempts to use it to load data into python.

[TomNicholas]

@abarciauskas-bgse actually see zarr-developers/zarr-python#1661