Upscale without needing to extract frames

[BankaiNoJutsu]

Hey!

Is it possible to upscale the frames from memory instead of writing to disk? I was thinking using image2pipe may be an option? (The main ESRGAN has a python example for video files), however the ncnn Vulkan implementation only supports images or directories inputs.

This should speed up the process and decrease read/writes on disk

Just wondering if possible, as this would be a nice improvement.

[ONdraid]

Hi @BankaiNoJutsu,

Is it possible to upscale the frames from memory instead of writing to disk? I was thinking using image2pipe may be an option? (The main ESRGAN has a python example for video files), however the ncnn Vulkan implementation only supports images or directories inputs.

I was also thinking about using the inference_realesrgan_video.py instead of the vulkan implementation. I'd like to do this in future.

This should speed up the process and decrease read/writes on disk

Reducing the number of reads/writes would be great, indeed. About the speed... I built reve for the purpose of upscaling large videos ( I currently use it to upscale anime episodes & movies to 4K ). Because I don't want the file size of the upscaled videos to be huge, I use HEVC x265 10bit efficient encoding, which unfortunately runs on cpu and is really slow ( but it's worth it ) and in my case, the encoding is slower than upscaling, so increasing the upscale speed would not have much effect on processing time.

But these are my limitations. E.g. someone could have faster cpu, and file size would probably not be important for someone who would want to use this tool for upscaling small videos. So adding more and faster encoding options and improving the upscale speed makes sense.

Oh, and I have noticed you made a fork, so feel free to contribute ;) If you want to talk, here's my discord: ONdraid#4920

[BankaiNoJutsu]

Hey @ONdraid,

Yeah, i'm also looking into achieving in memory processing for the frames. I'll check out some crates and libs i found. I can send them if you like.

I was also looking into modifying the vulkan-ncnn implementation of real-esrgan to enable direct input of a video file, and then process it with opencv or other lib, frame by frame. This would avoid the intensive disk reads/writes as well.

Yep, i understand you choosing libx265 for the output. I have been looking into adding other codecs indeed, with a check of ffmpeg, to validate it's support before starting the process. I plan to implement SVT-HEVC, SVT-AV1 and possibly some VAAPI hardware accelerated codec.

I'll finish up what's on my mind and do a pull so we can see what you want to merge.

Oh and i added you on discord also :)