How safe is it to open NWBs without a context manager?

[jonahpearl]

I notice in the documentation, a lot of the time files are read like this:

io = NWBHDF5IO("basics_tutorial.nwb", mode="r")
nwb_file = io.read()

as opposed to using a context manager. Is this best practice?

Specifically, is it possible to end up with a corrupted HDF5 file if the kernel crashes unexpectedly while reading from a file that's opened like this? Or is reading generally less prone to this sort of thing happening than writing? I tried doing this and restarting the kernel a bunch of times and it was fine, but it's hard to recreate an unexpected crash.

I ask because I'm trying to make it a bit easier to analyze data from many NWBs files at once in, say, a jupyter notebook, by collecting references to all the relevant NWB objects in some object (dict, DataFrame, etc), and then being able to do something like:

# store a reference to the data interface into a df
sessions_df.loc[sessn, 'interpolated'] = nwb_file.processing['behavior']['interpd_500']
...

# get it back out, programatically, without having to remember exactly where it's stored in the NWB
interpd = sessions_df.loc[sessn, 'interpolated']
data = interpd['signal'].data
...

You can also imagine this being useful to being able to subset data by various kinds of session / subject level metadata.

Other solutions to address this issue are also welcome. The other obvious way is to extract all the data from the NWBs into some set of linked dataframes and load it all into memory (a la the Allen SDK), but I'd like to avoid loading all the data into memory!

Thanks :)

[jonahpearl]

Ok, I wrote a small class to handle opening and closing the files politely. Curious to hear thoughts about whether there's a better way.

It's much slower than keeping the file handles open, but on my machine it still only takes ~100 ms to open and read each file. So even if you're reading 5-10 NWB fields and doing that for 5-10 sessions, it will still only take you an extra ~10 seconds, which feels tolerable. Of course if you're literally plotting raw data it will be painfully slow, but in that case, just read in the data you want to work with little chunks at a time.

class WorkerBee():
    """A small class to make accessing interface-level NWB data easier.
    """
    def __init__(self, nwb_path, nwb_field, path_to_interface):
        """
            nwb_path: path to the nwb file
            nwb_field: the top-level field of the NWB file to look under
            path_to_interface: a list of keys to access the desired behavioral interface
        """
        self.nwb_path = nwb_path
        self.nwb_field = nwb_field
        self.path_to_interface = path_to_interface
        
    @staticmethod
    def _dict_recurse(dict_, keys):
        if len(keys) == 1:
            return dict_[keys[0]]
        else:
            return WorkerBee._dict_recurse(dict_[keys[0]], keys[1:])
    
    # takes ~120 ms to open an NWB file handle and read a ~1 million-samples-long timeseries
    # takes ~5 us to actually read the data.
    # so this is a tradeoff between blazing fast code and not corrupting the data files!
    def get(self, timeseries):
        with nu.open_nwb(self.nwb_path) as nwb_file:  
            field = getattr(nwb_file, self.nwb_field)
            interface = self._dict_recurse(field, self.path_to_interface)
            ts = interface[timeseries]
            return np.array([ts.timestamps[:], ts.data[:]])
            
            
w = WorkerBee(
    nwb_path='/n/groups/datta/Jonah/Turbulator/Raw_data/2023-01-20_gmou85 copy.nwb',
    nwb_field='processing', 
    path_to_interface=['behavior', 'interpd_500']
)

# example of storing a worker bee in a dict
my_dict = {}
my_dict['my_favorite_session'] = {}
my_dict['my_favorite_session']['interpd'] = w

# get data from the worker bee
my_dict['my_favorite_session']['interpd'].get('therm_filtered')

[oruebel]

Specifically, is it possible to end up with a corrupted HDF5 file if the kernel crashes unexpectedly while reading from a file that's opened like this?

No, opening a file with mode 'r' enforces that no changes are being made to the file and as such the file cannot be corrupted.

as opposed to using a context manager. Is this best practice?

For read it is common because on read users often need to perform a large number of read operations on the same file that do not nicely fit into a context manager. To avoid opening and parsing the file for every read, the file is typically opened until the end of the analysis. The file should then be closed explicitly (note files are also closed automatically when the NWBHDF5IO object is being deleted, e.g., when the script finishes).

I ask because I'm trying to make it a bit easier to analyze data from many NWBs files at once in, say, a jupyter notebook, by collecting references to all the relevant NWB objects in some object (dict, DataFrame, etc),

Note, the files will need to remain open for the references to work. In particular, datasets are loaded lazily as h5py.Dataset objects in PyNWB, and so in order to read data, e.g,. from a TimeSeries object, the file will need to be open.

[jonahpearl]

Thanks for your help! I feel like I still don't totally understand the nuance. On the one hand

the file [opened with 'r'] cannot be corrupted

but on the other hand

The file should...be closed explicitly.

Why explicitly close the file if it gets closed automatically anyways? It probably helps with garbage collection and such in scripts, sure. But if I'm just running a notebook to do data analysis, should I always make sure to run a cell that closes all my open files before shutting it down?

[rly]

There are no super clear answers on this, but from what I can find, no, you do not need to close all of your HDF5 (NWB) files that are open in "r" mode before shutting down a notebook or python process. I don't do it. In non-notebook code, I would recommend using a context manager or explicitly calling close.

refs:

• comment from one of the primary maintainers of h5py https://groups.google.com/g/h5py/c/TM0W-0c90TY
• stackoverflow people https://stackoverflow.com/questions/56149237/do-i-need-to-manually-close-a-hdf5-file