Assigning starting_frames using VideoInterface in neuroconv

[SkepticRaven]

Hello!

I've been trying to adopt integration of our metadata into the NWB format using this tutorial: https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/behavior/video.html

The only major difference that I have with that tutorial is that we segment our data into multiple videos, so I provide a list of avi files to the VideoInterface() call.

After trying to add all the metadata in, I get an error when trying to call run_conversion()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<snip>/nwb.venv/lib/python3.10/site-packages/neuroconv/basedatainterface.py", line 99, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata=metadata, **conversion_options)
  File "<snip>/nwb.venv/lib/python3.10/site-packages/neuroconv/datainterfaces/behavior/video/videodatainterface.py", line 321, in add_to_nwbfile
    raise TypeError(
TypeError: Multiple paths were specified for the ImageSeries, but no starting_frames were specified!

How do I assign starting_frames, as I didn't see them in any of the schema (eg interface.get_source_schema() or interface.get_metadata_schema())?

I've tried interface.set_aligned_segment_starting_times(list_of_int_delta_times) and interface.set_aligned_starting_time(0.0), but was unsuccessful.

[CodyCBakerPhD]

starting_frames is exposed as a conversion option: https://github.com/catalystneuro/neuroconv/blob/main/src/neuroconv/datainterfaces/behavior/video/videodatainterface.py#L232

So in theory you should just be able to add

interface.run_conversion(nwbfile_path=path_to_save_nwbfile, metadata=metadata, starting_frames=[1.0, 2.0, 3.0, ...])

to that last line of the tutorial

I also forget if this is something changed since the last official release or not - if that doesn't work, try installing current main with pip install git+https://github.com/catalystneuro/neuroconv.git@main

We plan to add a feature that calculates this value automatically under the hood but we haven't gotten around to that yet

[SkepticRaven]

I was using the git pip version (for future people that find this).

Looks like that solution got past that issue and into another:

  File "<stdin>", line 1, in <module>
  File "<snip>/nwb.venv/lib/python3.10/site-packages/neuroconv/basedatainterface.py", line 99, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata=metadata, **conversion_options)
  File "<snip>/nwb.venv/lib/python3.10/site-packages/neuroconv/datainterfaces/behavior/video/videodatainterface.py", line 337, in add_to_nwbfile
    rate = video.get_video_fps()
  File "<snip>/nwb.venv/lib/python3.10/site-packages/neuroconv/datainterfaces/behavior/video/video_utils.py", line 62, in get_video_fps
    assert self.isOpened(), self._video_open_msg
AssertionError: The video file is not open!

It looks like I'll need to call interface.set_aligned_timestamps() to have it switch to timestamp mode (and not try and read the video not present). After that it works!

[SkepticRaven]

Sorry to get back to this a couple months later, but I think there's still a confusion with metadata assignment that I want to make sure this is operating as intended. Here's the current snippet that I'm working with:

from datetime import datetime
from dateutil import tz
from pathlib import Path
from neuroconv.datainterfaces import VideoInterface
import pynwb
import numpy as np

video_file_path = ['video_1', 'video_2']
interface = VideoInterface(file_paths=video_file_path, verbose=False)
interface.set_aligned_timestamps([[1.0, 2.0], [3.0, 4.0]])

metadata = interface.get_metadata()
# For data provenance we add the time zone information to the conversion
session_start_time = datetime(2020, 1, 1, 12, 30, 0, tzinfo=tz.gettz("US/Pacific"))
metadata["NWBFile"].update(session_start_time=session_start_time)
# Update some metadata
metadata["Behavior"]["Videos"][0].update(description='Video 1 of 2')
metadata["Behavior"]["Videos"][1].update(description='Video 2 of 2')

# Choose a path for saving the nwb file and run the conversion
path_to_save_nwbfile = 'test.nwb'
interface.run_conversion(nwbfile_path=path_to_save_nwbfile, metadata=metadata, starting_frames=np.array([0.0, 2.0], dtype=np.int64))

This is able to write a nwb file.

My confusion arises when the data is written out. The acquisition field only contains 1 video key and some of the data gets concatenated (timestamps, starting_frames) while other stuff gets discarded (comments, description).

with pynwb.NWBHDF5IO(path_to_save_nwbfile, mode='r') as io:
    nwbfile = io.read()
    print(nwbfile.acquisition)
    print(nwbfile.acquisition['Video: video_1'].external_file[:])
------
{'Video: video_1': Video: video_1 pynwb.image.ImageSeries at 0x139928555766736
Fields:
  comments: no comments
  conversion: 1.0
  data: <HDF5 dataset "data": shape (0, 0, 0), type "|u1">
  description: Video 1 of 2
  external_file: <StrDataset for HDF5 dataset "external_file": shape (2,), type "|O">
  format: external
  interval: 1
  offset: 0.0
  resolution: -1.0
  starting_frame: [0 2]
  timestamps: <HDF5 dataset "timestamps": shape (4,), type "<f8">
  timestamps_unit: seconds
  unit: Frames
}
['video_1' 'video_2']

Description field is now only the first videos description (second video description is discarded), but the original file list is placed into external_file for video_1.

I would have expected acquisition to contain a list of ImageSeries objects (1 per video with associated starting_frame and timestamps)). Is this how it's supposed to be functioning? So if I want to update the comments and description fields, I only need to modify the first in the metadata?

[CodyCBakerPhD]

@SkepticRaven The general rule of thumb for DataInterfaces in NeuroConv is that they produce a single neurodata object per interface. What you would expect in this case (external mode), which indeed seems to be occurring here, is the external_file field should be a list of the two videos in the order they occur as segments. As such if you expect multiple ImageSeries for each video (not recommended if they constitute the same contiguous video stream), you'd use separate VideoInterfaces for each.

As a general rule of thumb for neurodata objects in NWB, a single object will only have a single description. Sorry for the confusion about how you can set descriptions to different levels of the NeuroConv metadata and still have it be considered 'valid' but ignored. But suffice to say since we only have one object created through NeuroConv, you would only use the description of the first video index. It should also describe more about the video itself; the '1 out of 2' bit should be communicated by the length and order of the external_file list.

As far as comments not being propagated, I'll check into that. I would have thought it was allowed, same as description, again relative to first video index in the metadata.

1 per video with associated starting_frame

If an ImageSeries only had a single video in it, the starting_frame would not be useful; it is primarily used in conjunction with external mode to indicate the pre-calculated cumulative frame lengths of all prior videos (skipping the first since that's obviously 0)

[CodyCBakerPhD]

Can you show the code used to try to set comments to the interface?

I'm trying to reproduce and the field is showing up for me in the metadata schema: https://github.com/catalystneuro/neuroconv/blob/main/src/neuroconv/datainterfaces/behavior/video/videodatainterface.py#L48

which is then used to define each index like you're setting the descriptions: https://github.com/catalystneuro/neuroconv/blob/main/src/neuroconv/datainterfaces/behavior/video/videodatainterface.py#L48

and to clarify, only the first index metadata is used at this line: https://github.com/catalystneuro/neuroconv/blob/main/src/neuroconv/datainterfaces/behavior/video/videodatainterface.py#L325

(maybe we should add a warning or something if the other indices have attempted to set different values...)

But anything in that dictionary gets fed to the PyNWB object in the end: https://github.com/catalystneuro/neuroconv/blob/main/src/neuroconv/datainterfaces/behavior/video/videodatainterface.py#L429

[SkepticRaven]

I meant comments associated with videos past the first are discarded (similar to description in the example above).

# These fields are preserved
metadata["Behavior"]["Videos"][0].update(description='new description', comments='new comments')
# These fields are discarded, because they're not in the first index
metadata["Behavior"]["Videos"][1].update(description='new description 2', comments='new comments 2')

Thanks for confirming that it was doing what it's supposed to be doing (and not what I expected it to do). I saw in the metadata that each video had all the fields and assumed all those values would be propagated forward.