ndx-events for Matnwb?

[jccava]

Hello. I am trying to add the ndx-events extension to my Matnwb setup. I can successfully create the nwb file, but t crashes during nwbExport. I put the AnnotatedEventsTable into a processing module, as advised in the pynwb sample code for the ndx-events extension on GitHub, and the export dies trying to export that module. I have tried putting the AnnotatedEventsTable with in the nwdbdatainterface and in the dynamictable of the processing module. Should it work, or is Matnwb a little behind? Or could I just be doing it wrong? Should I try putting the AnnotatedEventsTable somewhere other than in a processing module? Thanks in advance for any insights.

[rly]

Could you copy and paste the error that you get here? The AnnotatedEventsTable should be fine to include in a ProcessingModule.

[jccava]

Thank you for the reply. If it's probable user error, I'll try a few more things before posting errors or code. I'm pretty new at this.

[rly]

This code snippet might help:

% create a ragged column to add to the annotated events table
bad_event_col = types.hdmf_common.VectorData( ...
	  'description', 'whether each event time should be excluded', ...
	  'data', [false, false, true, false, true] ...
);
bad_event_col_index = types.hdmf_common.VectorIndex( ...
	  'description', 'bad_event column index', ...
	  'target', types.untyped.ObjectView(bad_event_col), ...
	  'data', [3; 5] ...
);
annotated_events = types.ndx_events.AnnotatedEventsTable( ...
	  'description', 'annotated events from my experiment', ...
	  'colnames', {'bad_event'}, ...
	  'bad_event', bad_event_col, ...
	  'bad_event_index', bad_event_col_index, ...
	  'id', types.hdmf_common.ElementIdentifiers('data', [0; 1]) ...  % 0-indexed, for compatibility with Python
);
% place the annotated events table in a "behavior" processing module in the NWB file
behavior_mod = types.core.ProcessingModule('description', 'processed behavioral data');
behavior_mod.dynamictable.set('AnnotatedEvents', annotated_events);
nwb = NwbFile( ...
	  'session_description', 'mouse in open exploration', ...
	  'identifier', 'Mouse5_Day3', ...
	  'session_start_time', datetime(2018, 4, 25, 2, 30, 3) ...
);
nwb.processing.set('behavior', behavior_mod);

[jccava]

Thank you. Will get back to this on Monday and let you all know how it goes. I'll post my final code when I get it working.

[jccava]

Hi again. After some more messing around and a bit more learning, I think I can state my issue more intelligently now. On the ndx-events GitHub page there is a Python example. The snippet for AnnotatedEventsTable follows:

# create a new AnnotatedEventsTable type to hold annotated events
annotated_events = AnnotatedEventsTable(
    name='AnnotatedEventsTable',
    description='annotated events from my experiment',
    resolution=1e-5  # resolution of the timestamps, i.e., smallest possible difference between timestamps
)
# add a custom indexed (ragged) column to represent whether each event time was a bad event
annotated_events.add_column(
    name='bad_event',
    description='whether each event time should be excluded',
    index=True
)
# add an event type (row) to the AnnotatedEventsTable instance
annotated_events.add_event_type(
    label='Reward',
    event_description='Times when the subject received juice reward.',
    event_times=[1., 2., 3.],
    bad_event=[False, False, True],
    id=3
)

# create a processing module in the NWB file to hold processed events data
events_module = nwb.create_processing_module(
    name='events',
    description='processed event data'
)

# add the AnnotatedEventsTable instance to the processing module
events_module.add(annotated_events)

I am using Matnwb. My code looks like:

	pdgmEvents = types.ndx_events.AnnotatedEventsTable( ...
		'description', ['AET descrip: Events for ', pdgm],...
		'id', types.hdmf_common.ElementIdentifiers('data', []));
	pdgmEvents.colnames = {'evt', 'time'};


	for n = 1:length(dsTimingNames)
		pdgmEvents.addRow('evt', dsTimingNames{n}, 'time', double(trialVectorData{n}.data(:)));
	end

	eventsModule = types.core.ProcessingModule(...
		'description', [pdgm, ' events']);

	eventsModule.dynamictable.set('events', allEvents{d});
	f.processing.set(pdgm, eventsModule);

All is fine until nwbExport:

Error using types.ndx_events.AnnotatedEventsTable/export
Property `event_description` is required in `/processing/rfMap/events`.

Error in types.untyped.Set/export (line 181)
                    refs = v.export(fid, propfp, refs);

Error in types.core.ProcessingModule/export (line 82)
            refs = obj.dynamictable.export(fid, fullpath, refs);

Error in types.untyped.Set/export (line 181)
                    refs = v.export(fid, propfp, refs);

Error in types.core.NWBFile/export (line 1036)
            refs = obj.processing.export(fid, [fullpath '/processing'], refs);

Error in NwbFile/export (line 61)
                refs = export@types.core.NWBFile(obj, output_file_id, '/', {});

Error in nwbExport (line 35)
    export(nwb(i), filename);

Error in nwb_Events_test_0422 (line 373)
nwbExport(f, 'nwb_test_0228_file.nwb');

It is unclear to me from the Python example what is required. I can fill the AnnotatedEventsTable/event_description and event_times fields with arbitrary data to get farther, but it will then die on AnnotatedEventsTable/event_times_index. I suspect I am not using AnnotatedEventsTable:addRow correctly.

Thanks for your help on this.

[lawrence-mbf]

Looking at the ndx-events schema (https://github.com/rly/ndx-events/blob/main/spec/ndx-events.extensions.yaml), you need to include the following columns when calling addRow: event_times, label, and event_description which should all be generated properties of the class AnnotatedEventsTable.

Furthermore, event_times should be a cell array of double/single values such that addRow will auto-populate the (assumed) VectorIndex type event_times_index.

Hope that helps!

[jccava]

Thank you. Will get back to this on Monday and let you all know how it goes. I'll post my final code when I get it working.

[jccava]

Good morning. AnnotatedEventsTable.addRow is still unhappy with me.

        pdgmEvents = types.ndx_events.AnnotatedEventsTable( ...
            'description', ['AET descrip: Events for ', pdgm],...
            'colnames', {'paradigm'},...}
            'id', types.hdmf_common.ElementIdentifiers('data', [0:(length(dsTimingNames)-1)]));

        for n = 1:length(dsTimingNames)
            pdgmEvents.addRow('paradigm', pdgm, 'label', dsTimingNames{n}, 'event_description',...
                dsTimingNames{n}, 'event_times', double(trialVectorData{n}.data(:)));
        end

Error using types.util.dynamictable.addVarargRow
Invalid column name(s) { event_description, event_times, label }

Error in types.util.dynamictable.addRow (line 51)
    types.util.dynamictable.addVarargRow(DynamicTable, varargin{:});

Error in types.hdmf_common.DynamicTable/addRow (line 117)
        types.util.dynamictable.addRow(obj, varargin{:});

Error in nwb_Events_test_0422 (line 160)
            pdgmEvents.addRow('paradigm', pdgm, 'label', dsTimingNames{n}, 'event_description',...

The inputParser in the AnnotatedEventsTable constructor adds the parameters event_description, event_times, label. The inputParser in types.util.dynamictable.addVarargRow does not. Could this be the source of my error, or am I still doing something wrong in my call to addRow?

Thanks, James

[lawrence-mbf]

iirc, you do still need to add these columns to the colnames property even if they're supposed to be property names.

[jccava]

Thank you. Will try.