A Complete NWB File from a Pipeline: Extensions

[jmdelahanty]

Hello NWB!

I've planned out the kinds of things I'd like to include so we have a complete picture of our datasets that are created from the Bruker Ultima microscope. I wanted to open up a discussion here about extension writing for the additional things I wanted to include from the data I'm gathering during this project.

A complete NWB File for the Bruker 2P setup will include:
Raw Data

• Raw behavior data (voltage recording)
• Raw video of mouse face
• Raw 2P Data
• Raw Z-stacks (likely 8 - 16 of them) per indicator
• Arduino configuration
• Trial structure configuration
• Behavioral setup metadata
• Laser/PMT/Pockel/Scope objective configuration for T-Series
• Laser/PMT/Pockel/Scope objective configuration for Z-Stacks

Processed Data

• Reference image for recordings from Suite2P
• Suite2p configurations for segmentation/analysis
• Motion corrected t-series
• Cell registrations as optical-channels
• Cell traces
• Spike deconvolution (maybe...)
• Averaged z-stacks
• HOG Matrcies of mouse facial expression
• Mouse behavior timestamps
• More?

I'm writing in the discussion here because I'm not sure what should be included in an extension/how broad they should be. I'm planning on the following extensions from this list (which might be incomplete):

1. Behavior metadata: Include the Arduino metadata and trial structure metadata used when running the experiment
2. ImageSeries: Include Laser/PMT/Pockel/Objective information for the T-Series collected
3. Mouse facial expressions (so raw video, HOG matrix values, and emotion label")
4. Surgery: Additional information I'm collecting as part of a mouse's metadata for the project. Would be part of the Subject class.
5. Suite2P Configurations: I thought it would be cool if the values used for a given recording's motion correction and cell registration would be included in the file so people could run the analysis themselves later if they'd like.

The first things I'd like to add are the behavioral metadata and the Image Series metadata. Any advice for how I might actually build these extensions out? Thanks everyone!

Edit: The idea is that a base level NWB file is built the moment a recording is finished and written to the server for a given subject. This base file would have any and all possible metadata from the experiment included. I'd be appending the raw and processed data later since turning it into something that's not Bruker's proprietary format takes quite a long time per recording (45min-1hr). If possible somehow/if anyone has any time/desire to check it out, it would be immensely helpful to have someone see how I'm actually doing this since I'm the only person who has ever seen what I've written and I don't want it to suck.

[bendichter]

@jmdelahanty thanks for the detailed question. This is going to take a bit of time to go through. Hang tight.

[CodyCBakerPhD]

@jmdelahanty Looks like quite the undertaking! I can't wait to see what these look like...

Here are some suggestions for the things you had questions about:

1. What types of metadata does the Arduino system capture? Depending on complexity, I’d recommend utilizing the existing Trials Table https://pynwb.readthedocs.io/en/stable/tutorials/general/file.html#time-intervals wherever possible; if the setup requires significantly more than you think that structure can provide, a good guideline example of this sort of thing from past work would be the https://github.com/catalystneuro/ndx-tank-metadata project.

2. For ImageSeries, see if the existing module https://pynwb.readthedocs.io/en/stable/pynwb.image.html?highlight=ImageSeries can do everything you’d need. Also check the OpticalChannel and ImagingPlace objects here: https://pynwb.readthedocs.io/en/stable/pynwb.ophys.html#pynwb.ophys.OpticalChannel. Otherwise I’d only bother with a full extension here if there are things you would want to form custom reference links with other objects in the file, like the raw T-series.

3. These three things could certainly be written in tandem; raw video can be an ImageSeries, and the emotion label could utilize a very commonly used extension: ndx-events https://github.com/nwb-extensions/ndx-events-record. I’m unsure of the histogram gradient; would you want to store all such block values over each frame, or sliding windows (both more of a TimeSeries object at that point) or averaged/convolved from the raw video?

4. The API descriptor for session level metadata for the NWBFile can be extremely dense (https://pynwb.readthedocs.io/en/stable/pynwb.file.html#pynwb.file.NWBFile), but surgery info is actually attached at this level, rather than the Subject object.

5. Without an extension, adding this information would be done at the level of the ‘description’ field for the corresponding ImageSegmentation https://pynwb.readthedocs.io/en/stable/pynwb.ophys.html#pynwb.ophys.ImageSegmentation.add_segmentation. However, if that information is quite dense, this would be a good place to make an extension that basically extends the attributes of the ImageSegmentation class to contain Suite2p specific configuration values.

[jmdelahanty]

Hello Cody! Nice to e-meet you. Thanks so much for your thorough answers!

I think it might be a fair bit of work, but I'm really excited for this! Thankfully, my lab mates will all be working together to help make this kind of thing happen. Also, I believe that our lab will be working with you all at CatalystNeuro in the near future!

What types of metadata does the Arduino system capture?

The goal for this would be to have a complete set of information that's used to run the Arduino script that actually delivers stimuli for the experiments run under our Bruker microscope. There's an example here in the repository that actually runs things.

I thought it would be cool to have a way for people with their own Arduinos have a way that they could literally run the experiment if they wanted using the public repository and see the parameters that generated the trial structures used in the data.

Each experiment that's run generates a configuration file that contains the precise trial structure, duration of stimuli for each trial/ITI, any frames that were dropped, and other information about the session. It looks like ndx-tank-metadata is a good example.

For ImageSeries, see if the existing module can do everything you’d need.

Looking closer at the docs, it does seem like it does do what I'd need for this! For the description field, is it acceptable to have a sort of dictionary for any additional values? As in objective: Nikon 20x, ..., key:value?

These three things could certainly be written in tandem; raw video can be an ImageSeries, and the emotion label could utilize a very commonly used extension: ndx-events.

Okay! That sounds like it'd be great! I think we're pretty far away from actually implementing this step properly so I will keep this in mind as we move forward.

Surgery info is actually attached at this level, rather than the Subject object

Something I've been toying with is how I'd want to store the different kinds of brain injections I've been doing for our mice. Currently, I have a .yml file that contains subject specific information about all sorts of things. You can see an example of that here.

In order to use some parts of the system that run imaging experiments automatically, I need this kind of information available. For example, if you have multiple indicators and want to do a Z-stack for both of them, I need to know what laser wavelengths to use and what coordinates you injected into so I can programmatically build the NWB file. I thought it would be interesting to have all this sort of metadata available for a particular animal inside the NWB file so you can know precisely what happened to it for any given dataset you want to look at.

Without an extension, adding this information would be done at the level of the ‘description’ field for the corresponding ImageSegmentation

This relates to the ImageSeries piece above and has the same question: Can those parameters be encoded as key:value pairs in the description?

Thanks for your help everyone! I'm learning a ton and I'm having lots of fun doing it.

[CodyCBakerPhD]

Something I've been toying with is how I'd want to store the different kinds of brain injections I've been doing for our mice. Currently, I have a .yml file that contains subject specific information about all sorts of things. You can see an example of that here.

Oh, my! That is quite a bit more surgery information than I think I've ever seen before... That level of detail could certainly merit a custom extension for all of those attributes.

This relates to the ImageSeries piece above and has the same question: Can those parameters be encoded as key:value pairs in the description?

This pretty much sums up many of your remaining questions... For information of this level of richness, I would advise against writing key-value pairs in the description field; it's really meant to be a simple string of text that someone can quickly read to get context on the object.

For rich hierarchical parameter information like what you have here, I would indeed advise the work of creating an extension for whatever type of data object that happens to be, so that it includes its own schematic way of tracking the attributes for that special type of data. That way these attributes can also, themselves, contain simple string descriptions describing that field to someone reading an NWBFile written through the extensions, and understand what it means in the context of the object.

Looking forward to seeing some of these as they are created!

[jmdelahanty]

Oh, my! That is quite a bit more surgery information than I think I've ever seen before... That level of detail could certainly merit a custom extension for all of those attributes.

I'm not sure if it's really all necessary, but the data fields included in the example are what many people in the lab use to keep track of their surgeries. I'm working on a GUI called "Scalpel!" that could output that kind of information when a surgery is completed but it's taken a back seat to other things I need to work on. Our lab's AI engineer has suggested that we simplify where possible and that including all this in an extension could be useful but, for now, we should probably focus on other data specific fields.

We'll be keeping you posted as things move forward! And I'm hoping that we can work with you at Catalyst Neuro in the near future too!