where to put spike sorting analysis entities into NWB file

[magland]

I'd like to put the following entities into a NWB file, and I'd love to get advice on where they belong. The goal is to make a NWB file associated with a particular spike sorting output for the purpose of evaluating the units and comparing with ground truth when available.

• Spike trains for sorted units - I know where this goes
• Spike trains for ground truth unit(s) - I think this should be a separate "Units" object in /processing/ground_truth
• Autocorrelograms - A bunch of bins and counts (histograms). One histogram/autocorrelogram per unit
• Cross-correlograms - A bunch of histograms associated with pairs of units - (a sparse sampling of the pairs)
• Average spike waveforms - for each unit, a T x M array where T is the number of timepoints for the waveform, and M is the number of electrode channels in the neighborhood of the peak channel for the unit. The tricky thing is that M will vary depending on the unit. We'll also need to store the indices of the electrodes corresponding to the neighborhood.
• Spike amplitudes - an amplitude value for each event in the spike train for each unit
• Spike waveform snippets (subsampled) - for each unit, a K x T x M array where K is the number of subsampled events, and T and M are as above.

[CodyCBakerPhD]

Spike trains for ground truth unit(s) - I think this should be a separate "Units" object in /processing/ground_truth

Precisely

Autocorrelograms - A bunch of bins and counts (histograms). One histogram/autocorrelogram per unit
Cross-correlograms - A bunch of histograms associated with pairs of units - (a sparse sampling of the pairs)

Uncharted territory AFAIK - would a generic TimeSeries or DynamicTable structure be sufficient?

Average spike waveforms
Spike amplitudes - an amplitude value for each event in the spike train for each unit
Spike waveform snippets (subsampled) - for each unit, a K x T x M array where K is the number of subsampled events, and T and M are as above.

These are all properties of the units and so best belong on the same Units table (this is the current policy/suggestion at least)

Templates (both mean and standard deviation) have a place in the schema of each Units table: specifically waveform_mean and waveform_sd

Full waveforms also have a place: waveforms

Amplitudes would just be an extra custom column (since the Units table is just a table after all, it can have as many custom columns as you like, including potentially those correlograms you mention)

The only thing I would mention is the waveforms are typically quite large and as such you would need to add the column data in a safe/efficient way w.r.t. the data engineering. An example snippet of this that comes to mind for a separate table structure can be found here, though if you're source waveform array is a memmap and the slice over all units is too large to load into memory all at once you could additionally wrap that H5DataIO in a DataChunkIterator to write it iteratively

[magland]

Thanks. Regarding the /processing/ground_truth_units, how would I do that exactly using pynwb... for the standard /units I am doing:

nwbfile.add_unit(id=unit_id, spike_times=st)

How would I specify to add a unit at a different location.

Also I'm confused... by the above it seems like I am adding a "spike_times" entry in the row... whereas in the hdf5 file the spike_times is a single array with a spike_times_index. Is there something tricky internally that treats "spike_times" as a special argument?

[bendichter]

from pynwb.misc import Units
from pynwb.testing.mock.file import mock_NWBFile


nwbfile = mock_NWBFile()
nwbfile.create_processing_module("ecephys", "ground truth units")
units = Units("ground_truth_units")
units.add_unit(spike_times=[1,2,3])
nwbfile.processing["ecephys"].add(units)

nwbfile

[magland]

Thanks Ben, that worked! So now I have ground truth and the sorting output in the same NWB.

https://dandiarchive.org/dandiset/000618/draft/files?location=sub-paired-english

And in Neurosift you can see that ground truth corresponds with unit 37 in this sync'd raster plot

https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/333/51d/33351dba-31c0-450e-bbc4-d92cfa04cff8&tab=neurodata-items:view:DirectRasterPlot|/processing/ecephys/ground_truth_units@view:DirectRasterPlot|/units&tab-time=781.1288701619649,782.9678976827367,786.1075728698023

[bendichter]

impressive!

[magland]

I'm looking at the docs for pynwb.misc.Units and I don't see how to put in custom columns (e.g., spike_amplitudes)

https://pynwb.readthedocs.io/en/stable/pynwb.misc.html#pynwb.misc.Units

[bendichter]

from pynwb.misc import Units
from pynwb.testing.mock.file import mock_NWBFile


nwbfile = mock_NWBFile()
nwbfile.create_processing_module("ecephys", "ground truth units")
units = Units("ground_truth_units")
units.add_column("spike_amplitudes", "spike amplitudes", index=True)
units.add_unit(spike_times=[1,2,3], spike_amplitudes=[1,2,3])
nwbfile.processing["ecephys"].add(units)

nwbfile.processing["ecephys"]["ground_truth_units"].to_dataframe()

[bendichter]

index=True should be used when each cell of the table is a vector of variable length, as is the case here since amplitude is a feature of spikes, not of the unit. For a features of the unit like "brain_location", "firing_rate", or "mean_amplitude", you should not use index=True