xarray to_zarr creates group when var has / in the name, is this defined behavior?

[adair-kovac]

So say I create a dataframe with a data variable that has a slash in the name:

import xarray as xr, numpy as np
ds = xr.Dataset({"foo/bar": (("x"), np.random.rand(10))}, coords={"x": range(10)})
print(ds)

<xarray.Dataset>
Dimensions:  (x: 10)
Coordinates:
  * x        (x) int64 0 1 2 3 4 5 6 7 8 9
Data variables:
    foo/bar  (x) float64 0.3587 0.3285 0.8432 0.05657 ... 0.5356 0.4142 0.5616

Naively, I would expect to then be able to load it with open_dataset and get the same dataframe back. Instead, it's created a zarr group and the data is empty:

ds.to_zarr("foobar.zarr")
ds = xr.open_dataset("foobar.zarr", engine="zarr")
print(ds)

<xarray.Dataset>
Dimensions:  (x: 10)
Coordinates:
  * x        (x) int64 0 1 2 3 4 5 6 7 8 9
Data variables:
    *empty*

Of course I can load it with open_mfdataset passing two paths, but that still doesn't yield the original variable name:

ds = xr.open_mfdataset(["foobar.zarr", "foobar.zarr/foo"], engine="zarr")
print(ds)

<xarray.Dataset>
Dimensions:  (x: 10)
Coordinates:
  * x        (x) int64 0 1 2 3 4 5 6 7 8 9
Data variables:
    bar      (x) float64 dask.array<open_dataset-7ebb819920805b04e0a4921bf452...

I think what I'd expect or want to happen in this situation would be that I either get an error/warning when I try to write a variable with a slash in the name, or that it uses an escape character or something instead of writing a subgroup. Since my team is relying on this behavior now, though, my main concern is whether this is the defined/expected/intended behavior or something that might change in the future.

If it matters, my suggestion would be to retain this behavior but issue a warning that it's creating a zarr subgroup and that xarray doesn't support data hierarchies. The fact that xarray will write data hierarchies but has trouble reading them has been a problem for my team.

Edit:
For an example where it's a problem:

import xarray as xr, numpy as np
ds = xr.Dataset({"foo/bar": (("x"), np.random.rand(10)), "baz/bar": (("x"), np.random.rand(10))}, coords={"x": range(10)})
print(ds)

<xarray.Dataset>
Dimensions:  (x: 10)
Coordinates:
  * x        (x) int64 0 1 2 3 4 5 6 7 8 9
Data variables:
    foo/bar  (x) float64 0.4085 0.1657 0.8308 0.7351 ... 0.05793 0.3024 0.7001
    baz/bar  (x) float64 0.4878 0.7605 0.3839 0.2752 ... 0.162 0.9844 0.3285

ds.to_zarr("foobar.zarr")
ds = xr.open_mfdataset(["foobar.zarr", "foobar.zarr/foo", "foobar.zarr/baz"], engine="zarr")
print(ds)

ValueError: Could not find any dimension coordinates to use to order the datasets for concatenation

[adair-kovac]

@TomNicholas I'm addressing your comment on my HRRRZarr talk from AMS here for lack of a better discussion place.

xarray-datatree does read the HRRRZarr when using open_datatree, but the resulting datatree isn't useful since the hrrrzarr is suffering from the issue I documented above, which makes it essentially a malformed xarray dataset. The sort of hierarchy written by xarray as-is puts the coordinates at the root level but the data variable in a sub-subgroup, so reading it with datatree just leaves you with a root dataset with no data and a leaf dataset with no coordinates (example below).

While I would welcome an enhancement to datatree that supports inheritance of coordinates from a higher level in the hierarchy if that makes sense, it seems separate from datatree's intended use cases. (I actually don't really see why xarray couldn't be enhanced to read its own datasets of this format anyway, since the slash in the variable name is cosmetic and shows up in the .zmetadata.)

dt = open_datatree(s3fs.S3Map("hrrrzarr/sfc/20210101/20210101_00z_anl.zarr/surface/TMP", 
                              s3=s3), engine="zarr")
print(dt)

DataTree('None', parent=None)
│   Dimensions:                  (projection_x_coordinate: 1799,
│                                 projection_y_coordinate: 1059)
│   Coordinates:
│     * projection_x_coordinate  (projection_x_coordinate) float64 -2.698e+06 ......
│     * projection_y_coordinate  (projection_y_coordinate) float64 -1.587e+06 ......
│   Data variables:
│       forecast_period          timedelta64[ns] ...
│       forecast_reference_time  datetime64[ns] ...
│       height                   float64 ...
│       pressure                 float64 ...
│       time                     datetime64[ns] ...
└── DataTree('surface')
        Dimensions:  (projection_y_coordinate: 1059, projection_x_coordinate: 1799)
        Dimensions without coordinates: projection_y_coordinate, projection_x_coordinate
        Data variables:
            TMP      (projection_y_coordinate, projection_x_coordinate) float16 ...

[TomNicholas]

Hi @adair-kovac! Sorry you never got a reply to your original issue 😞

a data variable that has a slash in the name

I think what I'd expect or want to happen in this situation would be that I either get an error/warning when I try to write a variable with a slash in the name, or that it uses an escape character or something instead of writing a subgroup.

This is a good point, and I'm surprised we haven't dealt with this case before. At the very least the current behaviour (of treating names with slashes differently) should be better documented.

While I would welcome an enhancement to datatree that supports inheritance of coordinates from a higher level in the hierarchy if that makes sense, it seems separate from datatree's intended use cases.

I thought about this question a lot but the current plan is to not have any hierarchy - i.e. nothing inherited from nodes above.

I think what I'd expect or want to happen in this situation would be that I either get an error/warning when I try to write a variable with a slash in the name, or that it uses an escape character or something instead of writing a subgroup.

I'm not sure what the behaviour should be / is supposed to be. I think I have been building datatree assuming people wouldn't put slashes in their variable names (but didn't document that clearly).

@jhamman @shoyer what do you think?