How to save xarray.Dataset to groups in .NetCDF file

[joooeey]

From the docs, I was under the impression that this script to save to two different groups in the NetCDF file would work:

import numpy as np
from xarray import Dataset, DataArray

data_arrays = []
for i in range(4):
    data_arrays.append(DataArray(np.ones(3)*i))

ds1 = Dataset({"A": data_arrays[0], "B": data_arrays[1]})
ds2 = Dataset({"A": data_arrays[2], "B": data_arrays[3]})

ds1.to_netcdf("test.nc", "a", group="apple", format="NETCDF4")
ds2.to_netcdf("test.nc", "a", group="banana", format="NETCDF4")

Alas, it doesn't. I get the following Traceback:

Traceback (most recent call last):

  File "/home/lukas/Desktop/EDEO-Sensing/edeo_sensing/untitled0.py", line 19, in <module>
    ds1.to_netcdf("test.nc", "a", group="apple", format="NETCDF4")

  File "/home/lukas/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/xarray/core/dataset.py", line 1799, in to_netcdf
    return to_netcdf(

  File "/home/lukas/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/xarray/backends/api.py", line 1059, in to_netcdf
    store = store_open(target, mode, format, group, **kwargs)

  File "/home/lukas/anaconda3/envs/tf_gpu/lib/python3.9/site-packages/xarray/backends/scipy_.py", line 128, in __init__
    raise ValueError("cannot save to a group with the scipy.io.netcdf backend")

ValueError: cannot save to a group with the scipy.io.netcdf backend

How can I save my datasets to groups?

[keewis]

if I understand the error correctly, you're using the scipy backend which doesn't support saving to groups. To get this to work you should be able to specify engine="netcdf4" (assuming that netcdf4 is installed).

[joooeey]

Thank you for the quick response! Installing netcdf4 (with Poetry) indeed solved my issue.

[joooeey]

Neither specifying format nor engine is required as the docs state that netcdf4 is automatically used if it's available.

[jrmagers]

I noticed that when I do this:

ds1.to_netcdf("test.nc", "a", group="apple", format="NETCDF4")
ds2.to_netcdf("test.nc", "a", group="banana", format="NETCDF4")

I get an error:

FileNotFoundError: [Errno 2] No such file or directory: b'...\\test.nc'

But when I change the first save to write instead of append, it works:

ds1.to_netcdf("test.nc", "w", group="apple", format="NETCDF4")
ds2.to_netcdf("test.nc", "a", group="banana", format="NETCDF4")

I'd like to write a bunch of NetCDFs to different groups inside a single HDF5 file in a loop. Is there a more elegant way to handle this instead of having to make a "special case" out of the first write in order to create the file?

[keewis]

@jrmagers, can you open a new discussion for this question? I think that would make it a bit easier to answer, or convert to an issue if it turns out to be a bug.

[jrmagers]

see: #6564