Is there a way to open a NetCDF4 dataset that's split into subfiles in Parallel with Dask & Xarray?

[jacobgreen405]

I've been benchmarking NetCDF4 to Zarr format conversion times from randomly-generated datasets created with Xarray & Dask. Normally I'd just use xarray.open_mfdataset(paths, combine='by_coords', parallel=True), but I need to open this dataset from cloud storage. To do this for a single file, I use intake_xarray.netcdf.NetCDFSource(...).to_dask(), which works perfectly. This class does not seem to work for datasets comprised of subfiles, so I wrote a simple helper function to do this for me:

import xarray as xr
import gcsfs
import intake_xarray

def combine_nc_subfiles(base_uri, file, storage_options, filesystem):
    subfiles = filesystem.ls(f'{base_uri}/{file[:-1]}')
    uri_prefix = base_uri.split('//')[0]
    chunks = []

    for subfile in subfiles:
        subset = intake_xarray.netcdf.NetCDFSource(f'{uri_prefix}//{subfile}',
                                                  storage_options=storage_options,
                                                  chunks={}).to_dask()
        chunks.append(subset)

    return xr.combine_by_coords(chunks)


base_uri = "gs://my-bucket
file = path/to/nc_dataset/*"
storage_options={'token' : "/path/to/service-account-token.json"}

fs = gcsfs.GCSFileSystem(token=storage_options['token'])
ds = combine_nc_subfiles(base_uri, file, storage_options, fs)

The high computational cost of loading this way is seen by the Dask task graph given by the future returned during future = ds.to_zarr(store=f'{base_uri}/{upload_path}', storage_options=storage_options, consolidated=True, compute=False):

My question is, is there a better way to do this?

[dcherian]

I think the answer here is to use kerchunk. See also https://projectpythia.org/kerchunk-cookbook/README.html

[jacobgreen405]

From first glance, it looks like exactly what I need. Thanks for the reply!