Inconsistant number of frames after writing motion corrected video

[chrisda13]

Hi, I am trying to verify a few things with the pipeline and have a question about the video that is generated after motion correction.

I love the ability to motion correct based on a subset of the video. I have found that it works extremely well after the glow removal/denoise/background removal when I focus an a small subset of bright cells. My goal is to apply the shifts from motion correction to the original video (prior to the glow removal, denoise, and background removal steps) then use this motion corrected video as a baseline to compare various means of analysis (hand-drawn ROIs in ImageJ, minian, caiman, min1pipe, etc). I have been able to generate this video using the pipeline, and have been extremely happy with how stable the image is.

My steps are:
Load original video & specify coordinates for subset_mc
Run through glow removal, denoise, and background removal steps
Run estimate_motion & save the output
Restart the pipeline & load original video
SKIP glow removal, denoise, and background removal steps
Load motion that was saved out
apply_transform and save minian (creates Y_fm_chk and Y_hw_chk)
Write this motion corrected video to disk (not including the non-motion corrected video normally stitched) by running:
%%time
write_video(Y_fm_chk, "Y_fm_chk_vid.avi", dpath)

Currently, I am working with a subset of my data which is 4000 frames. When I open the data (saved out from the steps above) in ImageJ (after converting to raw format using ffmpeg), there are 4000 frames as expected. However, when I try to load the same video into minian or caiman, it is read as having 4800 frames. Do you know why this is the case?

I checked that Y_fm_chk had 4000 frames before saving it out, so I am not sure why it is 4800 when it is read back in (especially when it is 4000 in ImageJ). When I visualize the data, there is nothing obviously wrong (no blank or repeated frames). In fact the max/mean traces that are generated look nearly identical, except for the fact that one has 4000 frames and the other has 4800 frames. Any idea what is happening?

I'm more than happy to give you more info if that would help. Thanks in advance for the help!
-Chris

[phildong]

Hey so write_video is really supposed to write videos that're for visualization only, and the video might not be suitable for downstream analysis.
There's complication in video compression algorithms and distinction between "key frames", and as you noticed this may cause weird info in metatdata and may confuse a video reader.
If your down-stream analysis stays in python I'd recommend directly loading the binary data Y_fm_chk.
If you absolutely need the video as an intermediate, you'd probably need to hack the write_video function to make it write uncompressed avi.
I'd start trying to change vcodec='raw' here:

minian/minian/visualization.py

Line 1271 in a6d3339

.output(fname, pix_fmt="yuv420p", vcodec="libx264", r=30, **options)

Let me know if you're able to get it to work!

[svrks]

Hello, I was attempting to do this fix to get a raw video, but I get the following error downstream:

`BrokenPipeError Traceback (most recent call last)
File :2, in

File /data/user/home/savirkus/minian/minian/visualization.py:1276, in write_video(arr, vname, vpath, norm, options)
1268 process = (
1269 ffmpeg.input("pipe:", format="rawvideo", pix_fmt="gray", s="{}x{}".format(w, h))
1270 .filter("pad", int(np.ceil(w / 2) * 2), int(np.ceil(h / 2) * 2))
(...)
1273 .run_async(pipe_stdin=True)
1274 )
1275 for blk in arr.data.blocks:
-> 1276 process.stdin.write(np.array(blk).tobytes())
1277 process.stdin.close()
1278 process.wait()

BrokenPipeError: [Errno 32] Broken pipe`

My line 1271 looks just like above with the vcodec="raw" change incorporated. Any ideas how I can fix this? Thanks!

[chrisda13]

Thanks, I'll look into it at some point and see if I can get it to work.
One more question, is there a flag to fill in shifted pixels with something other than zero? Ideally, I would like to fill with the nearest value rather than 0. For example, if a frame is shifted down by 2 pixels, the top two rows are replaced with 0s, but can they be replaced with a replicate of the 3rd row? In the apply_transform function, there is an option: fill = 0. Is there a flag (something like fill = "nearest") that can be used to replace the data with the nearest non-zero pixel, or is this something I would have to change in the apply_transform function itself?

[phildong]

hey yes sadly you have to do a bit more hacking than changing that parameter but you're already on the right track!
Here is the line you'd need to play with:

minian/minian/motion_correction.py

Line 769 in a6d3339

fm = sitk.Resample(fm, fm, tx, sitk.sitkLinear, fill)

You'd probably want to use something like sitkNearestNeighbor instead of sitkLinear on that line. Some background here: http://insightsoftwareconsortium.github.io/SimpleITK-Notebooks/Python_html/21_Transforms_and_Resampling.html#Resampling
Let me know if you got it to work and/or if you want to put in an official feature request for this!

[hardawayja]

Hi,

We are interested in using the raw data after motion correction for analysis in ImageJ. @chrisda13 were you able to modify the write_video function to get the raw data as tiff or other format?

thanks,

Andrew

[Nate-the-Neuroscientist]

Has anyone been able to figure out how to write the raw videos after motion correction? Has anyone altered the code to produce this?

Like Andrew @hardawayja and Chris @chrisda13, I would also like to obtain the pre-processed and motion-corrected video before the video enters into the CNMF portion of the pipeline. I would like to be able to export the video without data loss at this step. For Chris, the motion-corrected video was less frames than the original video, but for me, the motion-corrected video is actually more frames than the original!

In my case, I want this functionality in order to apply the pre-processing steps of minian to epifluorescent imaging performed on brain slice cultures. For these experiments, the imaging interval is once every hour, the CNMF analysis (which depends on the presence of calcium transients in the data) would be irrelevant.