Best practices for overwriting + data exploration?

[jonahpearl]

Hi all -- two questions about best practices for working with NWB files.

1. I'm in the midst of setting up a new processing pipeline, and I'll often want to re-run an analysis that I just ran, e.g. after fixing a bug or changing a parameter. NWB files make this a bit difficult because HDF5 objects don't allow overwriting of existing data. I realize that the ideal way to do this kind of development is to do it outside of an NWB, and then move it in once it's solidified -- I largely do use that method, but 10% of the time, I need to make a change in code I've already written. The easy but slow solution is to just delete the NWB file each time; thankfully my pipeline only takes a few seconds. The better solution, I think, is to write a wrapper for things like add_timeseries(), which will check if the timeseries exists, and if it does, just overwrite the data attribute after checking the timestamps / size of data is the same. Is there a simple way I'm missing? Do others have better ideas about how to solve this?

2. Is there a method somewhere that will take a bunch of aligned time series in an interface and turn them into a pandas dataframe? It's not hard but it feels like something that probably exists already.

Thank you!

[bendichter]

1. Overwriting existing data is a complicated issue, and the best way to do it depends on what you are trying to change. In the above, you describe changing the values of an HDF5 dataset. This can be done pretty simply with PyNWB, e.g.:

nwb.acquisition['ElectricalSeries'].data[:3] = [1., .2, .3]

however this will only work if all you want to change is the values of the data and the data type and size are unchanged. If you need to change anything other than the data values, you'll need to do something else. You'll need to open the file with NWBHDF5IO in mode='r+' to have the ability the change the data.

2. We have a method for converting dynamic tables to pandas dataframes, but no built-in way to convert time series objects into one.