concatenating datasets along the time dimension when the time units change in each dataset

[rsignell]

I'm trying to use Virtualizarr to create references from multiple NetCDF files where each file's time variable has a unique units attribute.

To successfully build a virtual dataset that concatenates these files along the time dimension, I think I need to embed the decoded datetime values directly into the references, as a single dataset can only support one time unit, right?

If I were using Kerchunk, in MultiZarrToZarr I would specify:

coo_map={"ocean_time": "cf:ocean_time"},

but I don't know how to do this in Virtualzarr.

I could simply hard code in the time values, but this seems heavy-handed.

What is the best way to handle this?

[maxrjones]

The preprocess parameter is virtualizarr's equivalent to coo_map. You could open a feature request for a utility function to handle this, but for now I believe you would need to write a function that accomplishes the same as https://github.com/fsspec/kerchunk/blob/eff86485d8f46da8a6d83578359b5601841330c9/kerchunk/combine.py#L226-L252 and pass that as an argument to preprocess in https://virtualizarr.readthedocs.io/en/latest/api.html#virtualizarr.open_virtual_mfdataset.

See #669 (comment) for the best approach

[rsignell]

Okay, I'll give that a shot and report back!

[rsignell]

I'm wondering whether I might be able to avoid using preprocess by using loadable_variables?

It seems this might load the time values for each dataset as DateTime objects, which would be great.

But when I execute this:

ds_list = [vz.open_virtual_dataset(f's3://{url}', reader_options=ro, indexes={}, decode_times=True, loadable_variables={'time'})) for url in flist]

it just seems to hang (e.g. produces no errors, but never completes execution)

[rsignell]

Ugh, I had the loadable_variables as a dict, not a list. This works!

ds_list = [vz.open_virtual_dataset(f's3://{url}', reader_options=ro, indexes={}, decode_times=True, loadable_variables=['time'])) for url in flist]

combined_ds = xr.concat(
    ds_list, 
    dim='time', 
    coords='minimal', 
    compat='override', 
    combine_attrs='override')