Saving multiple xarray.Datasets to groups in a single HDF5 file

[jrmagers]

I'd like to write multiple xarray.Dataset's to different groups inside a single HDF5 file in a loop. Is there a more elegant way to handle this instead of having to make a "special case" out of the first write in order to create the file?

For example, when I do this:

ds1.to_netcdf("test.nc", "a", group="apple", format="NETCDF4")
ds2.to_netcdf("test.nc", "a", group="banana", format="NETCDF4")

I get an error:

FileNotFoundError: [Errno 2] No such file or directory: b'...\\test.nc'

But when I change the first save to write instead of append, it works:

ds1.to_netcdf("test.nc", "w", group="apple", format="NETCDF4")
ds2.to_netcdf("test.nc", "a", group="banana", format="NETCDF4")

This is akward when writing multiple groups in a for-loop since the first group written has to be written with "w" and all subsequent files written with mode "a".

[joooeey]

I don't have a solution. I'd just like to point out that this behavior is surprising because it's not consistent with the meaning of mode="a" in the built-in open function. I'd expect that this should create the file if it doesn't exist. Perhaps that warrants a change to the underlying Xarray code? Changing this would raise a backwards compatibility issue though.

For now, consider using try-except if that makes sense within the context.

[TomNicholas]

Support for multiple groups is one of the reasons why we are building xarray-datatree.

It's still just an early prototype, but it should already allow you to save multiple groups like this

In [1]: import xarray as xr

In [2]: ds1 = xr.Dataset({'foo': 0})

In [3]: ds2 = xr.Dataset({'bar': 1})

In [4]: from datatree import DataTree

In [5]: dt = DataTree.from_dict({'apple': ds1, 'banana': ds2})

In [6]: dt
Out[6]: 
DataTree('None', parent=None)
├── DataTree('apple')
│       Dimensions:  ()
│       Data variables:
│           foo      int64 0
└── DataTree('banana')
        Dimensions:  ()
        Data variables:
            bar      int64 1

In [7]: dt.to_netcdf("test.nc", format="NETCDF4")

>>> ncdump test.nc

netcdf test {

group: apple {
  variables:
  	int64 foo ;
  data:

   foo = 0 ;
  } // group apple

group: banana {
  variables:
  	int64 bar ;
  data:

   bar = 1 ;
  } // group banana
}

I don't know of a neater way to do this using only xarray.Dataset objects unfortunately - internally datatree is using the same trick you just suggested.

[jrmagers]

This looks like an elegant approach. Thanks!

[mronda]

@TomNicholas Does this work when trying to write multiple Datasets to same group?

[TomNicholas]

Not sure quite what you mean @mronda - in the DataTree model each Dataset corresponds to one group.

[mronda]

Apologies, @TomNicholas . I am curious if I can append to existing Group new Datasets given some dimension. Is this possible through that API?

[TomNicholas]

I'm still not quite clear on what you mean - I suggest you either try to write out what you want to achieve in code, or just have a go with DataTree. Feel free to raise questions on that repo too.

…

On Sat, 18 Jun 2022, 14:15 Maximiliano Ronda, ***@***.***> wrote: Apologies, @TomNicholas <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_TomNicholas&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=qdISi9HqjazmE0DcySuXts3OlnplnLfKjH4hpzAV0xo&m=p71stHI9yhbuVx5K669N5qC4GyxLtp7x49NLhJ1PcgM9FvMobbDaTdU2dUvlCjas&s=6WTNls8cIn72cyjt1GeL0oWDnwJpFzX9KnnfhnyPr_w&e=> . I am curious if I can append to existing Group new Datasets given some dimension. Is this possible through that API? — Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pydata_xarray_discussions_6564-23discussioncomment-2D2977637&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=qdISi9HqjazmE0DcySuXts3OlnplnLfKjH4hpzAV0xo&m=p71stHI9yhbuVx5K669N5qC4GyxLtp7x49NLhJ1PcgM9FvMobbDaTdU2dUvlCjas&s=eO3-KCDdVr6hkjNsXIYAOft5xQva8JweJvTym-hbivs&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AISNPI6LSPJEAWX3C7ZIBNDVPYG4ZANCNFSM5U4Y6IJA&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=qdISi9HqjazmE0DcySuXts3OlnplnLfKjH4hpzAV0xo&m=p71stHI9yhbuVx5K669N5qC4GyxLtp7x49NLhJ1PcgM9FvMobbDaTdU2dUvlCjas&s=v43pUshtu7rd3hNTRaEvJvngtR2JtGxSJxdKxiP8ovs&e=> . You are receiving this because you were mentioned.Message ID: ***@***.***>