Difference between various xarray methods

[PythonSchlumpf]

Hi,
I am a newbie with respect to xarray and I am looking for some help, exchange, guidance.
What is the difference between xr.open_mfdataset, xr.open_dataarray, xr.open_dataset, please?
I have not understand the difference.

Edit: Looks like xr.open_mfdataset can open multiple files, xr.open_dataset only one, but also xr.open_dataarray opens somehow one.
I tried to apply xr.open_mfdataset to my list of files, but I failed. They don't have dimensions or coordinates per se.

I have multiple hdf5 files ('.h5', created with h5py), each of them has multi-dim numpy arrays in them with a description of attributes. Example: a stack of 10 2d numpy arrays with low_idx=0, high=idx=9, another stack of 10 2d numpy arrays with low_idx=10, high=idx=19 etc, last file: 1 2d numpy array with low_idx=20, high_idx=20.
How could I go through 1000 hdf5 files, extract the numpy arrays, put them together in one location, and maybe have a dimension idx running from 0 to 9999?
Should I opt for xarray dataset, because I want to combine multiple information from other sources with it?
Thank you.

[TomNicholas]

Hi @PythonSchlumpf , sorry to hear that in this case the documentation didn't answer your questions.

Edit: Looks like xr.open_mfdataset can open multiple files, xr.open_dataset only one, but also xr.open_dataarray opens somehow one.

Are you comfortable with the idea that xarray provides multiple data objects you can use? Specifically a DataArray represents a single multi-dimensional variable (with optional coordinate variables stored alongside), and a Dataset represents effectively a dictionary of multi-dimensional variables (again with optional coordinates)?

The difference between xr.open_dataarray and xr.open_dataset is that whilst each will open a single file, the former will return only a DataArray, whereas the latter will return a Dataset. So if your hdf5 file contains multiple data variables which you want to acces, you should use xr.open_dataset.

I have multiple hdf5 files

You can open each file separately and combine them later (see the docs on combining xarray objects), but if your hdf5 files are related you likely want to open multiple files in one go, which is what xr.open_mfdataset is for.

How could I go through 1000 hdf5 files, extract the numpy arrays, put them together in one location, and maybe have a dimension idx running from 0 to 9999?

You can open them all in one go with open_mfdataset. If your files already have a coordinate in them called idx, then you can just do open_mfdataset(list_of_all_the_files, combine='by_coords') to automatically join them all up in the correct order. If your files don't already contain this information, you might have to arrange them yourself before opening with open_mfdataset(nested_list_of_files, combine='nested'), or even open them all separately and combine yourself.

I recommend you start by opening them one-by-one (so open each one with open_dataset) and combine them manually using some combination of concat, merge, combine_nested as described here. Then once you think you understand how you wish to combine everything it should be possible to open them all in one go with open_mfdataset.

1000 is quite a lot of files though - depending on how large they are you might find that you would get more performance by having a smaller number of larger files (which you could create using xarray.dataset.to_netcdf()).

P.S. If you have any ideas for how you think the docs on this could be improved then we're all ears!

[PythonSchlumpf]

Dear @TomNicholas
I am so grateful for your reply, some things have become a bit clearer. Thank you so much for your time. I will try to answer.

Are you comfortable with the idea that xarray provides multiple data objects you can use? Specifically a DataArray represents a single multi-dimensional variable (with optional coordinate variables stored alongside), and a Dataset represents effectively a dictionary of multi-dimensional variables (again with optional coordinates)?

My understanding has been that xarray has DataArrays. These are multi-dimensional arrays (like numpy arrays), where the labelled axes are called dimensions. The ticks on these axes are coordinates. I thought that I could combine several of such DataArrays in a Dataset. However, I did not understand the word "aligned" in "Dataset: A dict-like collection of DataArray objects with aligned dimensions." From: http://xarray.pydata.org/en/stable/user-guide/terminology.html

The difference between xr.open_dataarray and xr.open_dataset is that whilst each will open a single file, the former will return only a DataArray, whereas the latter will return a Dataset. So if your hdf5 file contains multiple data variables which you want to acces, you should use xr.open_dataset.

Thank you. First of all, I found out only later that there are other methods than xr.open_dataarray. This one was the first I encountered in the xarray manual. I managed with xr.open_dataarray to open 3 of my hdf5 files in a loop, for example. I think now, it was only a lazy load. In a second step, I learned how to rename the dimensions during the loop. Below my exploration:

print('File list:\n',fl)
sub_group='/entry/data/'
new_dims_names=('num_img', 'dim_1', 'dim_2') 
for fn in fl:   
    print('Current file: ', fn)
    with xr.open_dataarray(fn,chunks='auto', engine='netcdf4',group=sub_group, lock=True) as hf:
        #print(hf,type(hf))
        # We see for the first file:
        #Dimensions without coordinates: phony_dim_0, phony_dim_1, phony_dim_2
        #Attributes:
        #image_nr_low:   0
        #image_nr_high:  9
        #<class 'xarray.core.dataarray.DataArray'>
        print(hf.attrs)  #gets the attributes from the hdf5 file, example: {'image_nr_low': 0, 'image_nr_high': 9}
        print(hf.attrs.keys())  #get the keys of this hdf5 file
        print(hf.attrs.get('image_nr_low')) #it is a dict, let's get the values for the keys
        print(hf.attrs.get('image_nr_high')) #it is a dict, let's get the values for the keys
        print(hf.dims, type(hf.dims)) #what are the dims? type: tuple
        #if you want to rename the dimensions, override hf (Xarray's rename() returns a new Dataset object rather than modifying the an existing inplace.)
        hf=hf.rename({k:i for k,i in zip(hf.dims,new_dims_names)}) #rename dim, structure: old: new
        print(hf.dims, type(hf.dims)) 
        #assign new coordinates to num_img
        image_index=np.arange(hf.attrs.get('image_nr_low'),hf.attrs.get('image_nr_high')+1,1)
        hf=hf.assign_coords(num_img=image_index) #assign new coordinates to dimension num_img, coordinates are the image numbers
        print(hf,'\n',type(hf))
        print('------------------------------------------')

whereas the latter will return a Dataset

Would this be then a DataSet with only one DataArray ? The user decides here, how he wants to arrange the data, right? In this case, it would be still difficult for me to decide whether to opt for a DataArray or a DataSet.

So if your hdf5 file contains multiple data variables which you want to acces, you should use xr.open_dataset.

It is even more complicated. The data of experimental method A is distributed over many hdf5 files. The data of experimental method B over many other hdf5 files. And some auxiliary parameters in other hdf5 files. I think, this sounds like a DataSet, where I could have a DataArray based on A, a DataArray based on B, and maybe the auxiliary parameters as coordinates? What do you think, please?

You can open each file separately and combine them later (see the docs on combining xarray objects), but if your hdf5 files are related you likely want to open multiple files in one go, which is what xr.open_mfdataset is for.

Alright. In my first try, I looped through 3 hdf5 files of experimental method A using xr.open_dataarray . But I did not know how to combine them. Thank you for pointing this out. Maybe this would be also slow? Not sure how to do this yet for 1000 hdf5 files. Ideally, I want to parallelise this process later or speed up. This is why I opted for xarray as it has dask support.
Later, I discovered xr.open_mfdataset, but until today, I failed to open the 3 files, because in my folder was another hdf5 as master with a different structure. That there is even a xr.open_mfdataset, I found out via googleing and finding https://githubmemory.com/repo/pydata/xarray/issues/5585

Today I tried:

sub_group='/entry/data/'
ds = xr.open_mfdataset("path/to/data/data_*.h5", combine='nested', concat_dim='phony_dim_0', group=sub_group)

and this worked flawlessly for my 3 example hdf5 files. Not sure, how it will behave for 1000 files.
But (see below), looks like this was maybe the way to go? I got

<xarray.Dataset>
Dimensions:  (phony_dim_0: 21, phony_dim_1: 2167, phony_dim_2: 2070)
Dimensions without coordinates: phony_dim_0, phony_dim_1, phony_dim_2
Data variables:
    data     (phony_dim_0, phony_dim_1, phony_dim_2) uint32 dask.array<chunksize=(10, 2167, 2070), meta=np.ndarray>
Attributes:
    NX_class:  NXdata <class 'xarray.core.dataset.Dataset'>

Does this mean I have a Dataset where one DataArray is called data with dimensions phony_dim_0, phony_dim_1, phony_dim_2 ? And are all 21 numpy arrays in the correct order? This is very important for me.

How could I go through 1000 hdf5 files, extract the numpy arrays, put them together in one location, and maybe have a dimension idx running from 0 to 9999?

You can open them all in one go with open_mfdataset. If your files already have a coordinate in them called idx, then you can just do open_mfdataset(list_of_all_the_files, combine='by_coords') to automatically join them all up in the correct order. If your files don't already contain this information, you might have to arrange them yourself before opening with open_mfdataset(nested_list_of_files, combine='nested'), or even open them all separately and combine yourself.

"If your files already have a coordinate in them called idx," - No, they don't have something like this. Each hdf5 has as dimensions ('phony_dim_0', 'phony_dim_1', 'phony_dim_2') , so basically no dimensions, right? But can I still use combine='nested', concat_dim='phony_dim_0' in xr.open_mfdataset to achieve that the numpy arrays are in order from 0 to 21 in the DataArray of my DataSet, please?
I can only see: The first hdf5 file has as attributes {'image_nr_low': 0, 'image_nr_high': 9}, (the next one 10, 19, respectively, and the last one only {'image_nr_low': 20, 'image_nr_high': 20}. xr.open_mfdataset does not take attributes as concat_dim or combine parameter.
What do you think, please?
Is open_mfdataset not faster then looping through with xr.open_dataarray and xr.open_dataset, please?

I recommend you start by opening them one-by-one (so open each one with open_dataset) and combine them manually using some combination of concat, merge, combine_nested as described here. Then once you think you understand how you wish to combine everything it should be possible to open them all in one go with open_mfdataset.

Okay, I try the first part with open_dataset. I just need to find a way to verify the order in comparison to open_mfdataset. Maybe there is a method to check if both multi-dimensional numpy arrays are equal?
open_dataset does a real load, not just a lazy load? It also closes the file handle correctly?

1000 is quite a lot of files though - depending on how large they are you might find that you would get more performance by having a smaller number of larger files (which you could create using xarray.dataset.to_netcdf()).

Yes, that is true. I have a set where only 3 hdf5 files are available with 21 2d-numpy arrays, distributed as 10, 10, 1 2d arrays.

P.S. If you have any ideas for how you think the docs on this could be improved then we're all ears!
I apologise. I did not mean to criticise. On one hand, I struggle in general to read the Python manuals, but I learn best with examples, for example, see the one linked above. The entry for xarray.open_dataarray does not have an example like other entries at the bottom. I also wondered if the mentioned list of **kwargs is complete or are there more optional arguments available? I struggle also to know what values the **kwargs could have. Maybe this entry could also have a link to open_mfdataset?
I am also not sure what engine I should use for hdf5 files created with h5py. Do you have a recommendation, please?

Again, many thanks. I am really happy, I can exchange with somebody who has more experience than me. Have a good weekend.