Current performance profile of thumbnail making

[antonfirsov]

Opening this FYI to help keeping our performance strategy on the right track. /cc @JimBobSquarePants @br3aker

"Clinton's benchmarks" just like our own version of it is downscaling very large 13 megapixelish images to create 150px thumbnails. I ran a CPU samples profiling with the code in #1773.

With this original setup, everything else but Jpeg decoding is negligible today:

Changing the thumbnail size to 800px doesn't make big difference:

Expanded view of baseline decode:

Comparison to others on current master:

|        Method | maxDegreeOfParallelism |       Mean |      Error |   StdDev | Ratio | RatioSD |     Gen 0 |     Gen 1 |     Gen 2 |    Allocated |
|-------------- |----------------------- |-----------:|-----------:|---------:|------:|--------:|----------:|----------:|----------:|-------------:|
| SystemDrawing |                     20 | 2,843.8 ms | 1,483.1 ms | 81.29 ms |  4.17 |    0.12 |         - |         - |         - |        30 KB |
|    ImageSharp |                     20 |   681.9 ms |   112.6 ms |  6.17 ms |  1.00 |    0.00 | 1000.0000 | 1000.0000 | 1000.0000 | 1,237,325 KB |
|        Magick |                     20 | 1,538.9 ms |   820.7 ms | 44.99 ms |  2.26 |    0.05 |         - |         - |         - |       106 KB |
|   MagicScaler |                     20 |   394.7 ms |   133.9 ms |  7.34 ms |  0.58 |    0.01 |         - |         - |         - |       107 KB |
|    SkiaBitmap |                     20 |   904.0 ms |   353.9 ms | 19.40 ms |  1.33 |    0.02 |         - |         - |         - |       117 KB |
|       NetVips |                     20 |   382.0 ms |   712.9 ms | 39.08 ms |  0.56 |    0.05 |         - |         - |         - |       102 KB |

[antonfirsov]

Some low hanging fruits. Vector<float> powered code in Block8x8F:

[br3aker]

Two different 8x8 block multiplications? If it's related to spectral data downscaling by 0.125 - free win. edit: it is -250ms

Chasing pure C/C++ implementations with pure C#? Sign me in, this will be an interesting ride.

[br3aker]

@antonfirsov I have a question (conversion pipeline is complicated). Is it possible or I would rather say 'simple' to convert colors from jpeg colorspaces to supported colorspaces using integers, int16 specifically?

[antonfirsov]

float32 -> int16 conversion can be done for free for future use.

I don't follow what do you mean by this, conversion still needs conversion & and PackSignedSaturate isn't it?

Is it possible or I would rather say 'simple' to convert colors from jpeg colorspaces to supported colorspaces using integers, int16 specifically?

There are 3 challanges:

• Jpeg color space values data is stored as float, so you either either bulk-convert to int16 beforehand, or redefine the conversion methods to output stuff directly as int16 (or byte)
• Legacy runtimes, which only have SIMD support for float. This means that unless we are fine regressing them significantly, we need to bifurcate the pipeline to a float and an int16 path, which will complicate the codebase a lot.
• There are many colorspaces

Maybe if we come up with a modular design trick, that first implements the int16 path for YCbCr leaving the rest of the paths intact that can reduce the amount of the work.

---

That said, looking at the profiler data, I would rather start "integerizing" the pipeline from the other end:

While the current color conversion (ConvertToRgbInplace + NormalizedFloatToByteSaturate) costs about 2.3%, the floading point TransformIDCT costs 10.4% and the "round" in NormalizeColorsAndRoundInplace (7.62%) only exists to match libjpeg output better, which does a "rounding" (precision loss) as a side-effect of using integers-only.

We also have #1607 as a possible low hanging fruit for someone who understands the Huffman decoder better.

[br3aker]

I'm already working on decoder optimizations. Huffman decoder is definitely in my plans but it is really complicated, I expect it to be a full overhaul like I did it with the last encoder PR, so I saved the best for last :)

Jpeg color space values data is stored as float, so you either either bulk-convert to int16

That's what's almost free, convertion chain of float32 -> int32 -> int16 via sse/avx has 'built-in' rounding via cvtps_epi32 intrinsic. I've already optimized IDCT to use another implementation similar to FDCT one and this rounding stuff was close to it so that's where the question about int16 conversion came from. 2.3% is really to low to spend time on it for now.

"round" (...) only exists to match libjpeg output better

That's my main concern about decoding actually, what's the point of not only reducing precision but spending extra time on it? libjpeg-turbo (not sure about libjpeg) has 'legacy' floating point mode, int16 pipeline is used purely for performance. I would honestly vote to remove pre-rounding for performance reasons and get free quality out of it. What do you think about it? @JimBobSquarePants would be nice to have your opinion too :)

[antonfirsov]

I'm afraid that will quickly result in people opening issues like #245. We had some discussions about this in the past, I don't remember all details but what I concluded is that the closer is our output to libjpeg-turbo's, the better. Its a de-facto standard.

[br3aker]

While this is not a number 1 performance priority right now, I will do some research later. We have more precise both FDCT and IDCT now so I expect rounding to be less influential. Thanks for the explanation!

Edit: tested provided images with test code from #245 - it works the same with and without pre-rouding. Thing is, that user re-encoded existing jpeg, I don't know what encoder was used (link from the original post is not working) but if it was integer libjpeg - the difference was most likely caused by re-encoding degradation, not rounding difference with libjpeg.

[JimBobSquarePants]

If we can drop it, we should. We just need to test carefully.