xarray and h5py, hdf5plugin - first run works, second run crashes.

[PythonSchlumpf]

Dear all,

I have h5 files, probably created with h5py.

In a first step, I want to check if all h5 files contain data and what is the shape of the np.arrays inside.

An extraction of my code:

import xarray as xr
import os
import numpy as np
import h5py
import hdf5plugin
import h5netcdf
import xarray as xr

for iscan, basename in enumerate(file_list_full_path):
    #print(iscan)
    counter.append(iscan)
    assert os.path.isfile(basename)

    #access the sub-group
    sub_group='/entry/data/data'

    #read from file
    with h5py.File(basename,'r') as h5f:

        try:
            #try to access here the data
            data_shape=h5f[sub_group][()].shape

        # if key is not accessible, a KeyError is raised, we need to catch it
        #"KeyError: "Unable to open object (object 'data' doesn't exist)""
        # if this is the case, we replace data_shape as [-1,-1,-1]
        except KeyError as e:
            print(iscan, e)
            data_shape=(-1)*np.ones([1,3],dtype=int)
            #print(data_shape)
            print('Missing data was found in repetition {0}, file: {1}'.format(iscan,basename))

        #append to empty array
        shape_counter=np.vstack([shape_counter,data_shape])    

This works well. And no matter how many times, I execute the code in Jupyter, it works.
Now, I thought I want to use xarray to extract from all h5 files the np.arrays.
I use

ds = xr.open_mfdataset(file_list_full_path, combine='nested', concat_dim='phony_dim_0', group='/entry/data')
print(ds, type(ds))

First run works fine. Output:

<xarray.Dataset>
Dimensions:  (phony_dim_0: 26, phony_dim_1: 2167, phony_dim_2: 2070)
Dimensions without coordinates: phony_dim_0, phony_dim_1, phony_dim_2
Data variables:
    data     (phony_dim_0, phony_dim_1, phony_dim_2) uint32 dask.array<chunksize=(2, 2167, 2070), meta=np.ndarray>
Attributes:
    NX_class:  NXdata <class 'xarray.core.dataset.Dataset'>

When I execute the jupyter notebook again, it crashes. And in the top section of the code. For me, it looks like the hdf files were not properly closed by open_mfdataset.

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/tmp/ipykernel_12620/892729638.py in <module>
     22 
     23     #read from file
---> 24     with h5py.File(basename,'r') as h5f:
     25 
     26         try:

~/.conda/envs/pyfai-0.20/lib/python3.9/site-packages/h5py/_hl/files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, **kwds)
    505                                  fs_persist=fs_persist, fs_threshold=fs_threshold,
    506                                  fs_page_size=fs_page_size)
--> 507                 fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
    508 
    509             if isinstance(libver, tuple):

~/.conda/envs/pyfai-0.20/lib/python3.9/site-packages/h5py/_hl/files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    218         if swmr and swmr_support:
    219             flags |= h5f.ACC_SWMR_READ
--> 220         fid = h5f.open(name, flags, fapl=fapl)
    221     elif mode == 'r+':
    222         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5f.pyx in h5py.h5f.open()

OSError: Unable to open file (file close degree doesn't match)

I am uncertain what to enter as group information in xr.open_mfdataset. With h5py I need to provide /entry/data/data to make it work. Indeed, the path to the data in the .h5 is '/entry/data/data' and it exists when I check. But when I tested the xarray code section alone, it had worked. Anyways, I am confused now.
Could you help me to learn how to do it correctly, please? Thank you.

[PythonSchlumpf]

Okay, when I add ds.close() at the end, it seems to work in my shorter code version, but not in the longer. Strange. Could you maybe show me how to use xr.open_mfdataset with a context manager, i.e. the width statement, please? Could you maybe help me to understand better, why it has to be group=/entry/data/ ? And how to use it with h5py-generated files, please?
I saw as well now somebody used as option autoclose=True, what is this, please? But it does not seem to exist anymore?