[WIP] Add FlashVideo Text-to-Video Pipeline

[dg845]

What does this PR do?

This PR adds a pipeline for the FlashVideo (paper, code, weights) text-to-video model. FlashVideo is a two-stage text-to-video model based on CogVideoX that consists of a low-resolution video generation stage followed by a low-resolution to high-resolution upscaling model.

The FlashVideo model was requested in #10767.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@a-r-r-o-w
@yiyixuxu
@ghunkins

[a-r-r-o-w]

Thanks for working on this @dg845! Great to see you active again

Happy to review once you think it's ready & help with anything 🤗

[dg845]

Hi @a-r-r-o-w, do you think it would be better to have a single consolidated pipeline for both FlashVideo Stage 1 and Stage 2 inference, or to have separate Stage 1 and Stage 2 pipelines?

For context, the Stage 1 model generates low-resolution videos, whereas the Stage 2 model takes as input a Stage 1-generated low-res video and generates a high-res enhanced video from it. As far as I know, Stage 1 inference is essentially the same as CogVideoX1.0 inference (e.g. CogVideoXPipeline), while Stage 2 inference would be similar to CogVideoXVideoToVideoPipeline (for example, in how the latents are prepared). So for separate pipelines we could have a FlashVideoPipeline for Stage 1 models and a FlashVideoVideoToVideoPipeline for Stage 2 models; in the current code FlashVideoPipeline is trying to handle both Stage 1 and Stage 2 inference. (In the two pipeline case we could also consider whether a separate FlashVideo Stage 1 pipeline is necessary; I think it might end up being exactly the same as CogVideoXPipeline.)

[dg845]

Also @a-r-r-o-w, would it be possible to get GPU resources to help with testing the pipeline? My GPU (6GB VRAM) runs out of memory when naively trying to perform inference with the original code and I also get OOM errors when trying to load the larger Stage 1 checkpoint (based on CogVideoX1.0-5B) for e.g. diffusers checkpoint conversion.

For the memory requirements, I'm mostly thinking about the memory necessary to generate a video using the official code as reference for a slow test to check whether the diffusers implementation and original implementation are equivalent. It seems that at least 64GB of memory is needed currently for Stage 2 inference (see FoundationVision/FlashVideo#15).

[a-r-r-o-w]

do you think it would be better to have a single consolidated pipeline for both FlashVideo Stage 1 and Stage 2 inference, or to have separate Stage 1 and Stage 2 pipelines?

Let's do two pipelines given that each stage can be run on its own and Stage 2 does not seem to need an input that comes from Stage 1 (from a quick look, but I may be wrong).

Let's also just create a new FlashVideoPipeline instead of using the CogVideoX one for consistency despite the large amount of replicated code.

would it be possible to get GPU resources to help with testing the pipeline?

I think it should be possible for us to grant a GPU for awesome contributors like you but I don't know the exact details/process. Will cc @apolinario @linoytsaban @yiyixuxu for help. Inference will probably need atleast 40gb with the basic memory optimizations, but during integrations it's easier/faster to test with more memory available.

Really thanks a lot for taking this up! If you'd like me to run any conversion, please LMK and I'll upload the required files.

[a-r-r-o-w]

Let's rename this block to FlashVideoTransformerBlock

[a-r-r-o-w]

Let's remove the code for attention processors. Going forward, we will be adding support for this differently, and keep the implementations as minimal and only modeling-only as possible

[dg845]

To clarify, should the fuse_qkv_projections/unfuse_qkv_projections methods also be removed? And if so, should the corresponding pipeline methods be removed as well?

[a-r-r-o-w]

@dg845 Do you have an estimate of how long you'll need the GPUs for? @apolinario asked if you could create a HF Space. Once you do, an A100 can be allocated for the duration. Let us know if Spaces works for you or if you prefer something else

[dg845]

Hi @a-r-r-o-w, sorry for the late reply. I've worked a little with HF Spaces before and think it would work for me. IIRC stuff on HF Spaces needs to go through a gradio app so a normal cloud instance would probably be easier to work with, but I don't think this is a big deal. My estimate for how long I'd need the GPU would be a week (but may take longer). I also understand if this project is no longer a priority for resources.