How to modify an electrodes table?

[rly]

Summarized from a conversation I had with a user on Slack:

User: I want to add information to the NWB electrodes table after the file has already been generated.

Ryan: Do you want to add columns or modify data in existing columns?

User: Since the electrodes table requires a location column, our packaging code gives "unknown" for the whole location column. So we want to modify this column. In addition to that we want to add several columns that include the coordinate info of each electrode.

Ryan: To change the value of the location column, you can open the file in append mode and do nwbfile.electrodes.location.data[:] = new_numpy_array_or_list_with_strings
To add columns, you can do nwbfile.add_electrode_column(name="y", description="y", data=[2.0]) where data has the same number of rows as your table. then do io.write(nwbfile)

User: Right that part was clear. The alteration of location column was the tough part
After adding columns and altering location column, adding more rows shouldn't cause a problem right?

Ryan: That will cause a problem because, by default, tables are not expandable in that dimension. We are working on changing that default behavior but that feature is not available yet. So you will have to create a new electrodes table, replace the old one, and export the modified in-memory NWB file to a new file path.

User: There seems to be an add_row method?

Ryan: Yes, but once the datasets are written, they keep a fixed size. Using add_row on the electrodes table from an existing file would either raise an error or modify the objects only in memory; the changes would not be saved.

For more information, see this PyNWB tutorial on editing NWB files. https://pynwb.readthedocs.io/en/latest/tutorials/advanced_io/plot_editing.html#sphx-glr-tutorials-advanced-io-plot-editing-py
And on exporting files: https://pynwb.readthedocs.io/en/latest/export.html

[rly]

Related: How to add rows and columns to an already written DynamicTable (without export) NeurodataWithoutBorders/pynwb#1933

cc @rcpeene

[rly]

@rcpeene Note that the code in the linked issue will not work for the electrodes table because it contains a column "group" containing references to ElectrodeGroup objects, and appending to a column of references currently does not work. But we are actively working on this in hdmf-dev/hdmf#1135 (cc @mavaylon1)

[rcpeene]

Is this the explanation for the following error? I tried the recomendation above of using append mode and setting the locations column directly.

Traceback (most recent call last):
  File "/code/run_capsule.py", line 101, in <module>
    if __name__ == "__main__": run()
  File "/code/run_capsule.py", line 85, in run
    nwb.electrodes.location.data[:] = np.array(locs)
  File "/opt/conda/lib/python3.9/site-packages/zarr/core.py", line 1452, in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)
  File "/opt/conda/lib/python3.9/site-packages/zarr/core.py", line 1548, in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)
  File "/opt/conda/lib/python3.9/site-packages/zarr/core.py", line 1938, in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)
  File "/opt/conda/lib/python3.9/site-packages/zarr/core.py", line 1991, in _set_selection
    self._chunk_setitem(chunk_coords, chunk_selection, chunk_value, fields=fields)
  File "/opt/conda/lib/python3.9/site-packages/zarr/core.py", line 2259, in _chunk_setitem
    self._chunk_setitem_nosync(chunk_coords, chunk_selection, value, fields=fields)
  File "/opt/conda/lib/python3.9/site-packages/zarr/core.py", line 2269, in _chunk_setitem_nosync
    self.chunk_store[ckey] = self._encode_chunk(cdata)
  File "/opt/conda/lib/python3.9/site-packages/zarr/storage.py", line 1151, in __setitem__
    self._tofile(value, temp_path)
  File "/opt/conda/lib/python3.9/site-packages/zarr/storage.py", line 1111, in _tofile
    with open(fn, mode="wb") as f:
OSError: [Errno 30] Read-only file system: '/data/nwb/ecephys_session_experiment1_experiment_name.nwb/general/extracellular_ephys/electrodes/location/0.03c23a8f11ec48bea36ed2b233fa859a.partial'

[rly]

That sounds like a zarr-specific weirdness. I haven't seen a [x].partial file system before... I don't know why that would happen. Could you file an issue over in https://github.com/hdmf-dev/hdmf-zarr with example code that generates that?

[alejoe91]

@rcpeene the error is because in our casa the data folder is read only. You should first copy to results and then open in append mode ;)

[rcpeene]

Great. I believe that all worked. I now what I believe to be a separate issue exporting the electrodes table specifically with hdmf-zarr. It seems to fail to copy any of the original electrodes table. Issue reported here: hdmf-dev/hdmf-zarr#208