16:9 aspect ratio?

[james-s-tayler]

Since xformers support was added we can generate much higher resolution stuff now, I've been trying to take advantage of that and generate some 45 frame clips at 640x384 which is as close to 16:9 as it will let me set it and will successfully generate a clip without getting CUDA out of memory.

The problem I've been having with this is that the generations always contain either a subject that's duplicated or parts of their face will be duplicated. I'm assuming it's down to the model being trained on rectangular inputs and this duplication is just a particular failure mode owing to that. But, is it possible to do a finetune on 16:9 aspect ratio say? Or maybe there's a clever bit of prompt engineering where some One Weird Trick solves the problem?

[rookiemann]

Have you tried using the fine-tuned model?
https://github.com/deforum-art/sd-webui-text2video#fine-tunes-and-how-to-use-them

I too was having a lot of that problem with this doing my HD aspect as 448 X 256, until I switched out the model to the finetuned one it helps a lot. Sometime will still do it just a little but the overall results and the shutterstock watermark removal are good too.

[james-s-tayler]

I'm just using the original pruned weights. But that's good info.

[james-s-tayler]

Actually lowering the size to 448 x 256 itself helped somewhat. I think I'll take that as a win!

[kabachuha]

The equivalent of webui's hi. res. fix is planned #101

[james-s-tayler]

Oh man why didn't I think to loop it back through using vid2vid myself! I've been upscaling it by batch processing all the frames in img2img with restore faces enabled at a denoising strength of 0.3 and doing two rounds of that and it made it wayyyyyyy higher quality.

[james-s-tayler]

hmm I didn't have much luck with the vid2vid method. It came out looking the same?

With my method I'm using Chilloutmix as the model in img2img and doing 45 steps on DPM++ SDE Karras at 786x512 and a denoising strenght of 0.3

This was the first frame of "Katherine Heigl and James Marsden having a picnic in the middle of the road" out of ModelScope text2video at 384x256

This is after I run it through my img2img upscale method

So, when I use the batch processing for this it tends to upscale it pretty darn well and come out looking fairly awesome!

[james-s-tayler]

It doesn't fix every frame, some of them still have artefacts, but generally if I run this for 1 ~ 2 passes it makes the video pretty clear.