how to use to_zarr on a DataArray from stackstac?

[rbavery]

I can’t easily save an xarray DataArray to a Zarr. When I load a STAC catalog with stackstac

https://notebooksharing.space/view/d8e6268b7a122a4980322b8a69471056d0d90aa838475c986f12cd7c7f5135a3#displayOptions=

The above notebook gets a modest spatial subset for sentinel 1 rtc radar imagery and a pretty deep time series of 182 time points. It takes 1.5-2 minutes to load and materialize the DataArray on the Planetary Computer hub, so I’d like to save it to a Zarr for faster loading. The DataArray is called s1rtc_2016_2019

But after a few rounds of debugging, I can’t save this xarray DataArray to a Zarr

s1rtc_2016_2019.to_zarr("../data/s1rtc_2016_2019.zarr")
doesn’t exist, which I wouldn’t expect. only Dataset’s can be saved as Zarr.

So next I thought I could do a quick conversion from DataArray to Dataset and then save to Zarr.

s1rtc_2016_2019.to_dataset(name='s1bands').to_zarr("../data/s1rtc_2016_2019.zarr")

This still doesn’t work because there are non essential coordinates interpreted by stackstac that the xarray Zarr driver doesn't handle.

The error

ValueError: unable to infer dtype on variable None; xarray cannot serialize arbitrary Python objects

So I thought if I drop these coordinates, I'd be good. This still doesn't work because to_zarr doesn't like the empty attrs dictionary.

s1rtc_2016_2019.reset_coords(drop=True).to_dataset(name="s1bands").to_zarr("../data/s1rtc_2016_2019.zarr")

The error

TypeError: Invalid attribute in Dataset.attrs.

I'd like to understand, how do folks get around these errors when saving DataArrays to zarr? Has anyone saved a DataArray produced by stackstac to Zarr? And are xarray maintainers open to providing a DataArray.to_zarr method that uses sensible defaults and support and empty attrs dict or ignores attrs entirely? This would remove the three debugging steps I went through above to 1) convert the DataArray to a Dataset, remove coords with "arbitrary Python Objects", and the empty .attrs error.

[dcherian]

spec: RasterSpec(epsg=32722, bounds=(499980.0, 9417890.0, 514560.0, 9435480.0), resolutions_xy=(10.0, 10.0))

The error says this object is not serializable. Delete the "spec" attribute and it should work.

You may want to save this information somehow. I would ask over at rioxarray.

that uses sensible defaults

The defaults are sensible, though perhaps the error messages could be more useful (a PR to improve these is very welcome!). The software is forcing you to be explicit about what's written to disk, so you don't get surprised by missing variables or attributes when you read it back.

[rbavery]

thanks for the help @dcherian much appreciated

I'm not using rioxarray, but I'm using stackstac instead. I don't think deleting the "spec" attr solves this.

the "spec" attribute isn't present if I don't promote the attrs, which I tried. but I still can't save it to a Zarr because an empty attrs dictionary still results in the TypeError: Invalid attribute in Dataset.attrs. error

when I promote the attrs to try to solve the error that happens with an empty attrs

s1rtc_2016_2019.reset_coords(drop=True).to_dataset(name="s1bands", promote_attrs=True).attrs

{'spec': RasterSpec(epsg=32722, bounds=(499980.0, 9417890.0, 514560.0, 9435480.0), resolutions_xy=(10.0, 10.0)),
 'crs': 'epsg:32722',
 'transform': Affine(10.0, 0.0, 499980.0,
        0.0, -10.0, 9435480.0),
 'resolution': 10.0}

s1rtc_2016_2019.reset_coords(drop=True).to_dataset(name="s1bands").to_zarr("../data/s1rtc_2016_2019.zarr")

the error

TypeError: Invalid attribute in Dataset.attrs.

when I don't promote the attrs

s1rtc_2016_2019.reset_coords(drop=True).to_dataset(name="s1bands").attrs

s1rtc_2016_2019.reset_coords(drop=True).to_dataset(name="s1bands").to_zarr("../data/s1rtc_2016_2019.zarr")

TypeError: Invalid attribute in Dataset.attrs.

so the error is the same whether or not RasterSpec is present

[rbavery]

what did work was deleting the attrs with python objects but making sure there was at least one key in the attrs dict

del xarr.attrs['spec']
del xarr.attrs['transform']

maybe the error message in to_zarr could suggest deleting attrs with python objects and keeping at least one attr key? or there could be an argument in to_zarr that auto bypasses attrs that Zarr does not support? I think as a user, I'd prefer an argument to to_zarr that handles

1. bypassing attrs that Zarr doesn't support so that I don't need to delete offending attrs one by one
2. reset_coords to get rid of offending coords
3. does to_dataset

instead of raising a more informative error that still requires manually doing the above steps.

would the maintainers be open to a PR for a DataArray.to_zarr that accomplishes the above? It could be made explicit in the function name or description that this is for quickly saving and that metadata may be removed if it contains Python objects and is not supported by Zarr