FlashVideo <> Diffusers

[ghunkins]

Model/Pipeline/Scheduler description

There is a brilliant new two-stage video model released under the MIT license: FlashVideo.

Would be amazing to see this in Diffusers!

flashvideo.mp4

The left panel is the Stage I video (240p) and the right panel is the Stage II video (1080p).

This model will also be very useful for enhancing and upscaling videos generated via other pipelines (e.g. HunyuanVideo).

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Code: https://github.com/FoundationVision/FlashVideo/tree/main

Weights: https://huggingface.co/FoundationVision/FlashVideo

[ghunkins]

@a-r-r-o-w Any idea if this is a heavy lift? The authors say the architecture is heavily inspired from CogVideoX.

[a-r-r-o-w]

I looked through the code today and believe it should not require much work to port. Since it is based on the original CogVideoX codebase, all the conversion scripts to go from their format to our expected format should be re-usable: https://github.com/huggingface/diffusers/blob/d90cd3621dc9aea168ec928ff3aa9b977eb49c20/scripts/convert_cogvideox_to_diffusers.py

There are a few new modeling layers. What I'd recommend for porting is to copy over the CogVideoX transformer into a transformer_flashvideo.py and using the new layers where required, creating a new conversion script, and modifying a copy of the pipeline where necessary (should be minimal changes here I believe).

Please allow us some time for the integration until we find the bandwidth to take it up. Would be super happy to review and merge quickly if someone was open to PR it

[dg845]

Hi, I have started a draft PR for FlashVideo at #10838.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.