Benchmarking Specifications [Whiteboard]

[akai-katto]

I figured why use google docs to work on specifications, when there's a github discussion area.

Overview / Problem Statement

Currently dandere2x improvements have worked like this:

1. Find a video that does dandere2x preforms poorly on
2. Try to see which equations / formula's aren't working on that part.
3. Revise formulas + add new formula's sometimes
4. See if it fixed the problem
5. Run this new formula on a bunch of other sample videos.

Currently 5) is not automated at all / I have no reference of how a certain video will affect the visual fidelity of other videos, especially those of user videos. It's entirely possible for a theory to work on one video, but fail to generalize.

Proposed Solution

Using google collab / a local server, having an automated testing suite to upscale and process a variety of user videos, then preform PSNR metrics and sort them based on PSNR values, and I can manually review the worst-performed videos for a given branch. Doing this allows me to quickly (overnight) verify that a hypothesis generalizes or not to other styled videos, as it's been a common issue / slowdown for me to have to manually inspect each video after running dandere2x on it.

Use Case

Trying out new ffmpeg filters, having a "current branch" comparison to the next branch. This would also allow tracking improvements over time to be done a lot simpler, as I could compare branches from months ago to current branches.

[ghost]

Cool idea!! Would definitely help you find the "bad" code