OME Metadata for Microscopy

[jmdelahanty]

Hello everyone!

I'm currently trying to parse out our 2p imaging data's master OME-tiff file that comes out of Bruker's system and wondering how NWB handles OME metadata. Is there an example you have where I can see how people store OME formats/metadata in NWB alongside image series datatypes?

[bendichter]

Hey @jmdelahanty, we support automatic conversion of TIFF with this class that you'd use this like:

from roiextractors import NwbImagingExtractor, TiffImagingExtractor

imaging_ex = TiffImagingExtractor("imaging.tiff")
NwbImagingExtractor.write_imaging(imaging_ex, "output_path.nwb")

source

I don't know much about the OME component. @cechava has extensive experience with optical physiology and has offered to look into this.

[jmdelahanty]

Okay, great! I’ll be checking into that. Since the way Bruker gives tiffs is as individual files instead of stacks we’re creating looking into making HDF5 chunks after making multi page tiffs. It seems like MATLAB’s H5 writer can do this on its own, but in Python I haven’t found a way to make chunked data from individual tiffs yet.

…

________________________________ From: Ben Dichter ***@***.***> Sent: Tuesday, December 14, 2021 11:55:52 AM To: NeurodataWithoutBorders/helpdesk ***@***.***> Cc: Jeremy Delahanty ***@***.***>; Mention ***@***.***> Subject: Re: [NeurodataWithoutBorders/helpdesk] OME Metadata for Microscopy (Discussion #13) Hey @jmdelahanty<https://urldefense.com/v3/__https://github.com/jmdelahanty__;!!GX6Nv3_Pjr8b-17qtCok029Ok438DqXQ!m14omHY2jeMaFXFrS7Gra5wr9Ze6HcBfIw4e0j6V3yaMxn-H843TTBKK-DAJWC96PA$>, we support automatic conversion of TIFF with this class<https://urldefense.com/v3/__https://github.com/catalystneuro/roiextractors/blob/master/roiextractors/extractors/tiffimagingextractor/tiffimagingextractor.py__;!!GX6Nv3_Pjr8b-17qtCok029Ok438DqXQ!m14omHY2jeMaFXFrS7Gra5wr9Ze6HcBfIw4e0j6V3yaMxn-H843TTBKK-DBthTQYmw$> that you'd use this like: from roiextractors import NwbImagingExtractor, TiffImagingExtractor imaging_ex = TiffImagingExtractor("imaging.tiff") NwbImagingExtractor.write_imaging(imaging_ex, "output_path.nwb") source<https://urldefense.com/v3/__https://nwb-conversion-tools.readthedocs.io/en/master/part03_format_conversions.html__;!!GX6Nv3_Pjr8b-17qtCok029Ok438DqXQ!m14omHY2jeMaFXFrS7Gra5wr9Ze6HcBfIw4e0j6V3yaMxn-H843TTBKK-DBdOVTGag$> I don't know much about the OME component. @cechava<https://urldefense.com/v3/__https://github.com/cechava__;!!GX6Nv3_Pjr8b-17qtCok029Ok438DqXQ!m14omHY2jeMaFXFrS7Gra5wr9Ze6HcBfIw4e0j6V3yaMxn-H843TTBKK-DDv6ns1Lg$> has expensive experience with optical physiology and has offered to look into this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/NeurodataWithoutBorders/helpdesk/discussions/13*discussioncomment-1808265__;Iw!!GX6Nv3_Pjr8b-17qtCok029Ok438DqXQ!m14omHY2jeMaFXFrS7Gra5wr9Ze6HcBfIw4e0j6V3yaMxn-H843TTBKK-DD5jHCJPw$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AGYYKOIT7M3NJ2EX2J5K4CLUQ6OMRANCNFSM5J75EVIA__;!!GX6Nv3_Pjr8b-17qtCok029Ok438DqXQ!m14omHY2jeMaFXFrS7Gra5wr9Ze6HcBfIw4e0j6V3yaMxn-H843TTBKK-DDNiwp3Nw$>.

[bendichter]

h5py.File.create_dataset(, chunks=True)

But I don't think it's necessary to rewrite the data in a new format.

[jmdelahanty]

Thanks for the tip here again Ben! I'm looking into the Tiff Imaging Extractor now. Does this function basically read in a master tiff file and then load them all into an NWB file? We were thinking of linking the data to the NWB file and referencing things that way. To refresh, we're considering whether or not to basically write the tiffs directly into an NWB file and append to that all the intermediate steps as we move along or if we should just be linking to files on our filesystem for use later.

In the meantime out of curiosity, can we link to Zarr formats with NWB? When we start getting into these bigger datasets that might not fit in memory for processing having things stored in Zarr might be really helpful.

[oruebel]

When we start getting into these bigger datasets that might not fit in memory for processing having things stored in Zarr might be really helpful.

PyNWB and MatNWB both support lazy data load from HDF5, i.e., the APIs only parse the structure of the file whereas the data from the large datasets is being loaded on demand when you slice into the datasets. I.e., in terms of memory usage, it really depends on what parts of the data you select to load into memory. The storage format Zarr or HDF5 should as such not significantly affect the memory usage. For write, the NWB APIs also support iterative data write to support conversion of data that is too large to fit into memory. We have done some testing with Zarr as a backend as well hdmf-dev/hdmf#98 I'd be interested in learning more what your particular needs in this area are.

[jmdelahanty]

Hello @oruebel ! I remember the lazy loading piece of HDF5 which is really nice. I wish I understood more about how it's actually implemented! It makes sense that it won't make much difference on the memory usage side for reading the data/analyzing it iteratively.

I'd be interested in learning more what your particular needs in this area are.

We can hop on a zoom chat if you ever wanted to talk! It'd be cool to e-meet you.

I'm not sure I'd classify it as a real need, but it just seems that using Zarr allows for some lighting quick conversion/processing of files in parallel not only on CPU but also GPUs too! I'm still very much a newbie at Zarr/HDF5 and parallel processing, but it seems like we can do some pretty cool stuff using Zarr for processing. I was able to run a conversion from many tiffs into H5 (using MATLAB's h5create function in a script one of our lab's graduate students wrote) in about 25 minutes for around 47k tiffs without using compression. I still need to try out the TiffImagingExtractor but I haven't gotten to it yet. Although I need to see if it really did work, when using dask and zarr the conversion (which included using Blosc compression) took about 50 seconds! We don't really need to go from 25 minutes to 1 minute, but it's very cool that it's possible! It also reduced the filesize by about half. Again, going from around 45GB to 23GB isn't super necessary since we have tons of storage available for the group, but it's cool that it can be done.

What I think is interesting is that we can maybe execute lots of our pipeline that fast by using parallelization tools! It would make iterating with different kinds of settings for analysis super quick/easy to do for people. As datasets grow ever larger whether it be calcium imaging, voltage imaging, etc, having the option to run things in parallel by calling NWB files could be nice. My understanding is that there's not great ways to do similar speedups upon HDF5 files since they can't be written back to in parallel, but I could totally be wrong! Again, I'm super new to this so I'm likely missing things.