How to include ElectricalSeries/SpatialSeries in trials ?

[etiennedemontalivet]

Hi,

I am using a public dataset from which I have Spatial data ((x,y)-hand velocity) and LFP (spectral) features for multiple trials. I don't have continuous data, so I plan to store them under pynwb.file.NWBFile.trials like in the example code below. Is it the standard way of storing such data for trials ?

<summary>Code here:</summary>

import numpy as np
import pynwb
from datetime import datetime
from dateutil.tz import tzlocal

n_trials = 5
n_features = 3
n_channels = 10
n_times = 100

# create NWBFile
nwbfile = pynwb.NWBFile(
    session_description="test",
    identifier="test",
    session_start_time=datetime.now().astimezone(tz=tzlocal()),
)
# Electrodes
device = nwbfile.create_device(
    name="device",
    description="",
)
electrodes_group = nwbfile.create_electrode_group(
    name="electrodes_group",
    description="",
    device=device,
    location="",
)
for i_elec in range(n_channels):
    nwbfile.add_electrode(group=electrodes_group, location="somewhere")
all_table_region = nwbfile.create_electrode_table_region(
    region=list(np.arange(10)),
    description="all electrodes",
)
# Add trial columns then data
nwbfile.add_trial_column(name="velocity", description="velocity")
nwbfile.add_trial_column(name="features", description="features")
for i_trial in range(n_trials):
    # FEATURES
    features_nwb = pynwb.ecephys.FeatureExtraction(
        name="test_features",
        electrodes=all_table_region,
        description=[
            f"feature no {i_feat}" for i_feat in range(n_features)
        ],
        times=np.arange(n_times),
        features=np.random.rand(n_times, n_channels, n_features)
    )
    # VELOCITY
    velocity_nwb = pynwb.behavior.SpatialSeries(
        name="velocity",
        description="velocity",
        data=np.random.rand(n_times, 3),
        reference_frame="unknown",
        timestamps=np.arange(n_times),
    )
    
    nwbfile.add_trial(
        start_time=0.0,
        stop_time=1.0,
        id=i_trial,
        features=features_nwb,
        velocity=velocity_nwb,
    )
nwbfile

I also tried using Timeseries property of trials, but pynwb.ecephys.FeatureExtraction are not Timeseries.
Any help would be much appreciated !

[CodyCBakerPhD]

I would generally keep the trials table itself as light weight as possible and store the SpatialSeries and FeatureExtraction as separate objects under the processing module

You are right that then if you wanted to associate a single trial to a particular range of those other objects, you would use for example a TimeSeriesReference object pointing to the target you want you subset - and I do not know the FeatureExtraction object well enough to know if or how that type of reference would be possible. But I would also say that it's not required to indicate such a link

The philosophy being that 'trials' are just another data stream occurring during the session, and NWB containerizes all data streams related to the session, such that you can associate events between and across all the contents. Links between objects are, from that perspective, just there as 'helpers' or 'convenience' to draw attention or simplify indexing during analysis

[etiennedemontalivet]

The philosophy being that 'trials' are just another data stream occurring during the session

Does it mean that it is not relevant for non-continuous data with unknown time gaps? (such as multiple trials without the information of starting time of trial)

Or would you store the trial's data using lists ? like this:

velocity_x = []
velocity_y = []
features = []
for i_trial in range(n_trials):
    velocity_x.append(np.random.rand(n_times))
    velocity_y.append(np.random.rand(n_times))
    features.append(np.random.rand(n_times, n_channels, n_features))
    
velocity_nwb = pynwb.behavior.SpatialSeries(
    name="velocity_x",
    description="velocity",
    data=velocity_x,
    reference_frame="unknown",
    timestamps=np.arange(n_times),
)
features_nwb = pynwb.base.ProcessingModule(
    description="features",
    name="features",
    data_interfaces=[pynwb.ecephys.FeatureExtraction(
        name=f"features_{i_trial}",
        electrodes=all_table_region,
        description=[
            f"feature no {i_feat}" for i_feat in range(n_features)
        ],
        times=np.arange(n_times),
        features=np.random.rand(n_times, n_channels, n_features)
    ) for i_trial in range(n_trials)
    ])
nwbfile.add_acquisition(features_nwb)
nwbfile.add_acquisition(velocity_nwb)

[CodyCBakerPhD]

Does it mean that it is not relevant for non-continuous data with unknown time gaps? (such as multiple trials without the information of starting time of trial)

Wouldn't that mean that you also don't know the start_time of each row on the trials table anyway?

In general, unknown time gaps are not supported by core NWB. There are some extensions in progress that try to deal with this but none I know of that are popular or in common use yet

In general it also represents quite a significant loss of information that can impede other types of data reuse, one example being plasticity studies (looking at how the neural activity being monitored might change over time). Just knowing that trial index 5 occurred 'some time' after trial index 4, but with an unknown delay between them may not be sufficient in that case

Out of curiosity, is there a hardware constraint that would prevent this information from being recorded? Even if they are not perfectly aligned signals such as TTLs to a common clock, relying on something less precise like wallclock time would still be better than nothing

[etiennedemontalivet]

Does it mean that it is not relevant for non-continuous data with unknown time gaps? (such as multiple trials without the information of starting time of trial)

Wouldn't that mean that you also don't know the start_time of each row on the trials table anyway?

True.

Out of curiosity, is there a hardware constraint that would prevent this information from being recorded? Even if they are not perfectly aligned signals such as TTLs to a common clock, relying on something less precise like wallclock time would still be better than nothing

There is no hardware constraint, the information is (unfortunately) not in the dataset I am converting (from .mat files to NWB format) and I did not record this dataset myself.