img2video: I've a workflow which I think could be implemented but I don't know how to code

[jav12z]

The workflow is pretty simple indeed from a simple picture create a video, the video will have always the same frame ,and then leave vid2vid to do it's magic.

So this is my initial picture

From it I create a mp4 using ffmpeg

ffmpeg -loop 1 -i monkey.jpg -r 12 -t 4 -s 720x720 video.mp4

Now I feed this video to vid2vid and depending on the denoising strength and the seed I will obtain different results
Prompt: A monkey singing in a rock concert

https://imgur.com/TgdC0cl

https://imgur.com/QDp1Hwo

So I'm obtaining somekind of img2video less agressive than the inpaint solution.

Other examples using a single picture

https://imgur.com/a/WnlbOno

This can be expanded using different frames in the original video, vid2vid will try to do the interpolation, for example this was done with 3 pictures of ryu. From second 0 to 1 the first, from 1 to 2 the second, and from 2 to 3 the fourth the script used was

ffmpeg -loop 1 -t 1 -i ryu2.jpg -loop 1 -t 1 -i ryu3.jpg -loop 1 -t 1 -i ryu4.jpg -filter_complex "[0:v]fps=12[v0];[1:v]fps=12[v1];[2:v]fps=12[v2];[v0][v1][v2]concat=n=3:v=1:a=0" -r 12 -s 720x720 -t 3 video.mp4

https://imgur.com/lrf1G6Q

A better example of this workflow would be this one using frames from a set of heads in different positions

https://imgur.com/a/kLSFtvY

I think that a similar solution could be implemented to the extension.

Many thanks for this awesome extension.

[B34STW4RS]

Yeah, I been doing the same thing since about a couple days ago but by a slightly different method. I found that if I set the start frame to 1 in vid2vid, it would almost always produce video without the watermark, like 9/10 times no watermarks at all at any denoise level.

Now if only we could automate a process in which, the prompt gets sent to text2img first, then into a img2img loopback, and finally into vid2vid...