Different size in trial row for single column

[ahabdallah27]

I am adding trials and one of the columns can have a different length at each row. This seems to be causing a problem when I try to write the file because if I replace each row with the same data for that specific column, then I do not get the error when writing. Any thoughts on how to address this?

I am also curious about creating a timestamp. My data start time is 2 and end time is 1742172. I looked at tutorials that showed creating a list from the start time to the end time was needed. I don't understand the difference between data and timestamp so I am populating them with the same information. When writing the file, using the commented out portion my computer runs out of memory and I have to restart it. The uncommented portion works, but I don't believe it's correct. Am I setting up my time series wrong? I have 1341 trials of varying lengths of time. My understanding is that I am supposed to put all of the times of the trials in a single timestamp based on the feedback I got here: #69.

[CodyCBakerPhD]

I am adding trials and one of the columns can have a different length at each row.

This sounds like you want that column to be what's called a ragged array; this is accomplished using the index keyword argument when defining the trial column using add_trial_column (or add_column if directly constructing a DynamicTable) - you would set index=True for vector valued entries on each row, index=2 for matrix valued entries on each row (the dimensions do not need to be consistent over rows either)

I am also curious about creating a timestamp. My data start time is 2 and end time is 1742172. I looked at tutorials that showed creating a list from the start time to the end time was needed. I don't understand the difference between data and timestamp so I am populating them with the same information.

For trials, you are better off using the trials table rather than a TimeSeries - this table would have two required columns start_time and stop_time, whose values should be in units seconds relative to the session_start_time of the file. From the value of 1742172 it sounds like your time is currently an integer and might be in milli or microseconds (or even unix time)?

More generally, a TimeSeries is meant to represent some scalar, array, or matrix-valued change over time $\textbf{f} (t)$ and as such the first dimensions of data should be equal to the number of 'frames' acquired, which should either be regularly sampled (and using a scalar rate field to define the time basis) or irregular with the time points set as the vector of timestamps. Common examples of this include an ElectricalSeries which is voltage-valued and two-dimensional, with the second dimension being the number of channels or electrodes used to record neural activity.

[ahabdallah27]

I am trying the DataChunkIterator and it's taking quite a while to save the data. I stopped the write at 10 minutes because I don't think this is working as expected, but please do let me know if that's expected. My ExperimentTime and data fields each have almost 2 million elements. One is 2 million x 96 so there is a lot of data. This is how I'm doing it:

RawInputValsTimeSeries = TimeSeries( #preferably SpatialSeries, but needs reference frame
    name="Raw Finger Values",
    data=DataChunkIterator(data=np.array(RawInputVals)),
    unit="V",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

FingerAnglesTIMRLTimeSeries = TimeSeries( #preferably SpatialSeries, but needs reference frame
    name="Finger Angles",
    data=DataChunkIterator(data=np.array(FingerAnglesTIMRL)),
    unit="Decimal Percent",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

NeuralFeatureTimeSeries = TimeSeries(
    name="Spiking-Band Power",
    data=DataChunkIterator(data=np.array(NeuralFeature)),
    unit="Hz",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

SampleWidthTimeSeries = TimeSeries(
    name="Width of Spiking-Band Power",
    data=DataChunkIterator(data=np.array(SampleWidth)),
    unit="Count",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

DecodeTimeSeries = TimeSeries( #preferably SpatialSeries, but needs reference frame
    name="Decoder",
    data=DataChunkIterator(data=np.array(Decode)),
    unit="Decimal Percent",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

behavior_module = nwbfile.create_processing_module(
    name="behavior", description="processed behavioral data"
)
behavior_module.add(FingerAnglesTIMRLTimeSeries)
behavior_module.add(NeuralFeatureTimeSeries)

nwbfile.add_acquisition(RawInputValsTimeSeries)
nwbfile.add_acquisition(SampleWidthTimeSeries)
nwbfile.add_acquisition(DecodeTimeSeries)

with NWBHDF5IO("myfile.nwb", "w") as nwbfileio:
    nwbfileio.write(nwbfile)

[CodyCBakerPhD]

The DataChunkIterator performance is sensitive to two keyword arguments - iter_axis (which should be the longer of the two, and defaults to 0, so you should be OK there) and buffer_size (in units of 'number of elements to load along the iter_axis; all other axes fill to the maximum). If you're set on using the DataChunkIterator, try increasing buffer_size to 100_000 or so and see if that helps any

Since all your data seem to be numpy.array types (and hence sliceable), I'd strongly recommend giving the SliceableDataChunkIterator a try

from neuroconv.tools.hdmf import SliceableDataChunkIterator

which should be perfectly interchangeable with what you have currently written, except that it handles the buffering pattern with more grace and intuition, and its keyword arguments are all easier to understand (since they are written in terms of total RAM usage, etc)

[ahabdallah27]

I used it and was able to write the file, but it's taking up almost 2GB. Is there any compression with these methods? I can't find documentation for SliceableDataChunkIterator. I searched for it on https://neuroconv.readthedocs.io/en/main/ and couldn't find anything.

[rly]

Yes, you can compress the data. You would wrap the iterator with an H5DataIO and pass compression parameters. See "Step 2" here for an example using a different kind of iterator: https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/plot_iterative_write.html#sphx-glr-tutorials-advanced-io-plot-iterative-write-py

The key line is:

data2 = H5DataIO(data=matrix2, compression="gzip", compression_opts=4)

where you would replace matrix2 with your SliceableDataChunkIterator.

[ahabdallah27]

Thanks, it's much smaller now, (less than 300 mb)!