Hybrid Video Input in 2D/3D Mode

[reallybigname]

Hi everybody! @Funofabot encouraged me to introduce myself.

I created a mod for the Deforum notebook that composites video into normal 2D/3D animation mode. It has different modes for compositing:

• None: With no composite mask, it just does an alpha blend with video.
  • There is also an alpha schedule. The composite alpha affects the overall mix, whether you are using a composite or not.
• Blend: Uses a blend of video and prev_img to create a grayscale composite mask. There is also a blend schedule for blend amount.
• Difference: Uses the difference between video and prev_img to create a grayscale composite mask.
• Depth: Uses the depth map from the last frame as the compositing mask.
• Video Depth: Uses a video depth map from the current frame of video input as the compositing mask.
• There is also invert mask option, a mask contrast schedule, mask equalization options (Before, After, Both), and a mask auto-contrast feature which has high and low cutoffs - particularly useful for depth masks.
• I also have a color_coherence = "Video Input", which uses each frame of video as the color sample for the render.

After I made the compositing, I wanted more. When you mix video stuff and there's motion, things tend to stick to the screen rather than the environment. So, I developed some code for copying motion from video and applying the motion to the prev_img to prime the next frame:

• RANSAC: I have two different modes that use RANSAC to extract an Affine or a Perspective matrix from the video. Affine is just translation, scale, rotation. Perspective is like Affine, but it adds skewing (quasi-3D).
• Optical Flow: I have three optical flow modes using different cv2 optical flow implementations (Farneback, DenseRLOF, SF). They get an optical flow map and apply it to the prev_img.
• Motion modes can be used WITHOUT compositing video. You can just steal motion from a video.

I have a pull request in for the main Deforum notebook. But, I am now actively working on getting my same stuff into this webui version of Deforum, and I should have a PR soon! I have the UI stuff all done, my code copied into functions, and I'm just fixing a few minor incompatibilities.

DISCO Diffusion has a RAFT implementation of optical flow, and this can bring us a lot closer to that functionality. It's still way better for doing video input than Deforum, simply because of optical flow. And, even with my mods, RAFT is still a little bit better. But, RAFT takes some significant memory, as it is AI-based. All of my motion functions are part of cv2, which is already included code, and relatively light on impact. I may try to do a RAFT implementation in the future, but I would want to preprocess the flows, then clear that model from memory for the render. The way my stuff works is that video gets broken up into jpgs at the start, but the mixing and depth and motion stuff is all done at runtime, so it resumes easy, and doesn't have to save lots of files beforehand, or build big data structures for all frames. Although, I do have an option to save all of the extra images it makes during the process, composite masks, prev_img, and even a visualization of the optical flow with colors and arrows.

I'm happy to contribute to this awesome project!

I've worked hard to try to make my stuff compact and have no impact on other code. So, hopefully ya'll will adopt it. I've been a maverick, on my own, so far. But, I'll try to lend a hand where I can as part of a team.

If you want to see some of the various experiments I've been doing, check out my 🎬 YouTube channel, especially some of the more recent videos (although I have a few recent experiments with img2img and depth model on there - ignore those)

Here's one video to get you started on the type of things my code can do:
📺 Deforum Hybrid Video Experiments - EPIC FPV Drone One Shot in the Bar

[reallybigname]

#198

[ghost]

Thank you very much for this feature. I tried it, I can't get results as smooth as videos on your youtube channel but we can clearly see the possibilities it offers.
For my part, I lose the coherence of my init_video after a few frames. The camera movement remains faithful but the images diverge too much maybe I'm using the thing badly.
A step-by-step tutorial would be really great to properly understand all the parameters of this feature... if you can of course !

[reallybigname]

There are so many possibilities on how to use it! But, I do want to add some recommended settings to try out in the doc maybe.

I think a good place to start is to realize that if you set hybrid composite true, but leave comp mask as None, and set comp alpha to 1, it's essentially the same as normal Video Input mode, except for optional color matching and the number of steps taken depending on strength settings.

I would say to leave comp mask on None and ignore all the mask options until you've experimented with regular compositing with alpha.

I find a good place to start is to set strength to 0.75 or moew and comp alpha to 0.75 and see what it does. I regularly run at strength 0.9-0.95 though, and increase steps to like 400 (which is only actually taking like 20-40 steps.

PS: I do a little post-processing on my videos! A little flicker control in Vegas Pro goes a long way.

[ghost]

Thanks @reallybigname for your detailed answer, will test it for sure with these settings asap.

We had the opportunity to discuss with @Funofabot in a previous post, i am one of the users who love to test and give their feedback but i will be unable to make such tools. I discovered the real use of masks only recently so... !

I will go deeper into your program. But just the fact of being able to "copy" any camera movement from any video is already huge and it works pretty well. Can't wait to see the next updates of this stuff ! Bravo

[ghost]

@reallybigname I keep testing your feature which i like more and more :)

1-2x-RIFE-RIFE4.0-50fps.mp4

I think i have found some good settings, there are still problems with consistency with background, but i think it is irremediable when using videos with black backgrounds.
Btw i believe your feature adds a huge plus to the Deforum extension, hope you'll keep updating it!

[reallybigname]

Beautiful! I'm so glad to see some of the first videos made with these tools. I'm working on a bunch more optical flow methods!!!

Thanks!

[reallybigname]

I also have a new option that calculates motion differently... Instead of comparing the last video frame to the current video frame to get the motion, it compares the previous rendered image to the current video frame to get the motion. And, the option is a single checkbox, and it changes the behavior of ANY motion mode! It's trippy because it tried to lock the rendered frame to the incoming video frame, but if your settings make the rendered frame way different, the lock can get weird... but, it can also produce cool effects because of it!

[ghost]

@reallybigname Is this new option a game changer in term of tracking or result is just different ? Can not wait to see it !
I did several tests on the influence of masks, I can post the results here for those interested, if it can save time for some people.

Here are 3 different renderings. Only the "hybrid_comp_mask_type" setting has been changed for each one.
Strength : 0.68 / Color coherence : Video Input / hybrid_comp_alpha_schedule : 0.94 / hybrid_motion : Affine / hybrid_flow_method : DenseRLOF

hybrid_comp_mask_type : Depth

HybridDepth.mp4

hybrid_comp_mask_type : Video Depth

HybridVidDepth.mp4

hybrid_comp_mask_type : Difference

HybridVidDif.mp4

hybrid_comp_mask_type : Blend

HybridVidBlend.mp4

hybrid_comp_mask_type : None

HybridNone.mp4

For now and for me the best "consistent results" are obtained with "None" & "Difference" settings

[reallybigname]

Wow, these are fantastic! Great work.

Yeah, the usage all depends on what you're doing. I do a lot of stuff set on None, without the mask options. But, I also do a lot on Video Depth, with the high or low adjusted for he scene. I do a lot of stuff at strength 0.9 - 0.95 with steps way up to compensate, then bring the alpha down until you start getting a cool effect.

But, if you want to make an abstract thing that just uses the video to drive the animation indirectly, comp blend can produce some cool stuff (and has it's own mask blend alpha schedule). And, mask inversion, mask contrast schedule, equalization setting, and mask autocontrast low/high cutoffs can accomplish all sorts of things, depending on which mask you're using. With Video Depth, you can make waves of effect move to and from the camera by animating the cutoffs!

[reallybigname]

I just ordered a 4090.

[reallybigname]

Oh, regarding optical flow... Well, I had three styles, but due to incompatibility issues with contrib and normal opencv, only Farneback works for most people. So, I found some other ones that perform similarly. I have them working, but they also have settings that I'm messing with to try to give us a suite of different optical flow styles depending on usage. Some are better for fast motion, some for slow. Some better for camera movement, some tear more, etc...

So, you might see Farneback Slow, Farneback, Farneback Fast, for instance. Trying to find some good presets for people to use.

[ghost]

@reallybigname Nice ! I'll check this for sure. Regarding the 4090, it is a great investment when you work on AI stuff. The renderings above took about 20min each with this card.

However don't forget to make some modifications in the venv folder by replacing certain dll in "lib" because SD is not yet completely optimized for this card which can do even better and can be faster. Here is a link to have the best results for now, it worked for me : https://www.reddit.com/r/StableDiffusion/comments/y71q5k/4090_cudnn_performancespeed_fix_automatic1111/

[Funofabot]

THIS IS BADASS!

[markitzeroo]

@reallybigname looks awesome!

For some reason I can't get it to work with video input though. It seems to me that it only works in 2D & 3D animation mode (at least that what it says in automatic1111 gui), did I get this right?

[reallybigname]

It is an alternative to Video Input.

[Humanoidme]

would you say it captures more data from referenced video prior to generation than the standard video input?