Add minimal requirements — launching on 4 GBs of VRAM

[ClashSAN]

(Windows) (NVIDIA: Ampere) 4gb makes 192x192 videos, which are not bad!

[hithereai]

Indeed not bad. Which minimal requirements you wanted to discuss?

[ClashSAN]

192x192 is very coherent, 128x128 (doesn't do anything.)
there is no current advantage between newer and older gpus except for gpus handicapped to run at full precision.

• current minimal requirements: All 6gb fp16 capable gpus should be able to generate 192x192 videos.

[michaelbogdan]

On my Nvidia 2060 Super with 8GiB of VRAM I got video generation with 24 frames and 320x320 resolution to work at half precision, 360x360 fails with out of memory error. So for that resolution I think this is the minimal requirement.

[kabachuha]

I have a GPU with 4 gb VRAM, I'll check if it works

[kabachuha]

Idk why, but that button unloading still leaves a considerable amount of VRAM, which is preventing me from launch the model on your described resolution. Do you have any idea what module it could be?

[ClashSAN]

EDIT: @kabachuha the latest text2video_pytorch_model is the problem. If you use the text2video_pytorch_model from this commit, you will have no issue. https://huggingface.co/damo-vilab/modelscope-damo-text-to-video-synthesis/tree/7ecc3083c7f3b4214f2dbf61cbf300281f0f7b67

(Remove certain extensions.) I noticed the with aesthetic gradients extension active, the button did not allow me to unload the model completely. I was creating some recently.

Running modelscope would continue until it would OOM during vae decode phase.

Idk why, but that button unloading still leaves a considerable amount of VRAM, which is preventing me from launch the model on your described resolution. Do you have any idea what module it could be?

If I generate with sd, then unload, I see the residual vram as you have, but regardless of that I can still generate at 192 afterwards. I launched webui with a fp32 1.5 .ckpt but the others should work.
Try launching with --lowvram mode just ensuring it works for you at 192? It looks like this is possible with a 4gb 980 which is good to see: #35

Generate SD image, then do modelscope

After modelscope

DRAM

[kabachuha]

Still not worked even with the previous model.

I suspect it might be that the model is not being loaded fully in the memory before doing 'half' on it. I'm pruning the models to fp16 right now and will write if it helped. Pruned weights will be nice to have anyway to save the community's bandwidth

https://huggingface.co/kabachuha/modelscope-damo-text2video-pruned-weights

---

Well, now, it at least got past the first stage and broke on diffusion. I guess, some progress 🙂

[ClashSAN]

I do many consecutive generations, to get a feel for prompts, its consistent for me, just running with no flags on a very fresh installation and 9ef779c, used all your pruned weights.

Do you have Hardware Scheduling Disabled?
Testing with Windows 11 GeForce RTX 3050 Laptop GPU

other:
disable browser hardware acceleration
Go in nvidia control panel, 3d parameters, and change power profile to "maximum performance" (dunno if i did)

plugged in
Driver Version 31.0.15.2698

I have the bloated system compared to W10 and linux, so maybe its your gpu having slightly less vram somehow?

Well, now, it at least got past the first stage and broke on diffusion. I guess, some progress 🙂

What do you mean? it loads the model, then it drops?

[kabachuha]

I have the just same GeForce RTX 3050 Laptop GPU. Maybe, it's my KDE Linux setup that's breaking the processes to protect my system from overload? 🤔

I'll try running it on Windows

[diStyApps]

Thank you, works on 970

[kabachuha]

Omg

[diStyApps]

I didn't know github got 10mb limit of video, anyway a quick demo on 970 and 3080 on different resolution and lengths
https://www.youtube.com/watch?v=2tHn7lTcGR4

[balisujohn]

I can confirm this works with the half-size pruned models on a 1070ti on Ubuntu 22.04 with 8 gigs of vram, in case anyone has a similar card.

[kabachuha]

8 gigs of vram

Well, that is not low :)

The question was more about the minimal ones, such as only 4 GBs of VRAM

[JD1234JD1234]

i cant run 192x192 with 4GB vram 1050ti, 128x128 and 128x192 will run but give random/glitch patterns