Decode images directly to a target size?

[Webreaper]

I was using ImageSharp for bulk thumbnail generation, but ended up switching to SkiaSharp, because the performance of ImageSharp was just not quite quick enough. Loading a bunch of 7MB JPEGs, and then resizing down to 3 small thumbnails, takes around 1.2s per image with ImageSharp, and around 0.6s per image with SkiaSharp.

I know the bulk of the time is in the loading, and that you guys are working on intrinsics, which should speed things up significantly. But I wondered whether you also have the concept of reduced loading by codec on the roadmap? So with SkiaSharp, I can do this:

  SKCodec codec = SKCodec.Create(fileName);
  SKBitmap bmp = SKBitmap.Decode(codec, nearest);

where nearest is a scaled rect that I'm trying to load the optimal image for. The codec then loads just the size of image that's closest to nearest, which is significantly faster. This means that if I have a 4000x3600 image, and I want to resize to 800x600, Skia will load a 1000x800 image, which I can then resize. This clearly saves a lot of effort, and probably accounts for a lion-share of the processing time.

Any thoughts to add this to ImageSharp? Since so many people use it for Thumbnail generation/image reduction, it feels like this would be pretty welcome.

If ImageSharp can do this already, please let me know - I couldn't figure out how from the docs. :)

[antonfirsov]

@Webreaper this is a feature that currently does not exist, however I personally really would like to investigate and hopefully realize it for the next major release. However, I think the main performance benefit that comes with this is reduced memory footprint rather than better speed / lower CPU usage.

1.2s per image with ImageSharp, and around 0.6s per image with SkiaSharp.

That sounds a lot, should be much less for for both libraries normally. Can you elaborate a bit more on your use case? (What platform/runtime/hardware? What exactly is included in those 3 operations? How do you benchmark? Any sample (pseudo) code for the benchmark?)

[Webreaper]

Sure. I originally asked this question and it was the answer which lead me to think that loading that's the bottleneck.

My app is running in .Net Core 5-RC1, but I see the exact same perf characteristics in 3.1. It's running on a Mac (during development) and on a Linux box when deployed. My pipeline does the following in ImageSharp:

1. Loads the image
2. Orients it
3. For 3 thumbnail sizes, it resizes the image, optionally cropping (e.g., the smallest thumb is cropped to a square).
4. Save to disk

The 3 resizes are done in serial - so the first resize might be from 4000x3600 => 800 x 600, the next one would be 800x600 => 400x250, and the next 400x250 => 120x120. I use the previously resized image as the source for the next-smallest thumb to save time.

The images being converted are large (~7MB) images from an SLR camera. The app processes images in batches of 100, and typically takes around 120-130s for each 100 images.

            using var image = Image.Load(source.FullName);

            image.Mutate(x => x.AutoOrient());

            foreach (var pair in destFiles)
            {
                var dest = pair.Key;
                var config = pair.Value;
                var mode = ResizeMode.Max;

                var size = new Size { Height = config.height, Width = config.width };

                Logging.LogTrace("Generating thumbnail for {0}: {1}x{2}", source.Name, size.Width, size.Height);

                if (config.cropToRatio)
                {
                    // For the smallest thumbs, we crop to fix the aspect exactly.
                    mode = ResizeMode.Crop;
                }

                var opts = new ResizeOptions { Mode = mode, Size = size };

                // Note, we don't clone and resize from the original image, because that's expensive.
                // So we always resize the previously generated thumb, which will be faster for each
                // iteration because each previous image is progressively smaller. 
                image.Mutate(x => x.Resize(opts));
                image.Save(dest.FullName);
            }

I haven't profiled in detail; I can add some GetTickCount timings around the ImageSharp load and transform code, and do the same for SkiaSharp to figure out if one particular area would be the bottleneck. If you think it's possible to get this faster, then I'd much prefer to use ImageSharp - because the above code is so much simpler and more intuitive than the equivalent SkiaSharp code (which runs to about 300+ lines...).

[JimBobSquarePants]

I'd love to see this finished so we can improve our current performance further.

#1242

[Webreaper]

I just span it up compiled with .Net Core 3.1, and it's no faster....

I added a quick GetTickCount and the load takes an average of 1006ms for 200 images, and the thumbnail gen (for 3 thumbs) takes 657ms (average for 200 images). The AutoOrient call takes around 150ms.

That's consistent regardless of 3.1 or 5.0-RC1, seemingly.

[antonfirsov]

@Webreaper I think there are some factors about your scenario we are still missing. The 1.2 ms time is still too much for a typical dev machine, or at least it can not be explained by Jpeg decoder perf alone.

I tried to replicate a simplified variant of your scenario (no Auto-Orient, no file streams, focus only on Decode+Resize+Encode perf) with the following, 5.5MB 4752x3168 large test image:
https://github.com/SixLabors/ImageSharp/blob/master/tests/Images/Input/Jpg/issues/issue750-exif-tranform.jpg

With the following BenchMarkDotNet benchmark code:

public class LoadResizeSaveSeries
{
    private readonly Configuration configuration = new Configuration(new JpegConfigurationModule());
    private byte[] sourceBytes;
    private byte[] destBytes;
    private string TestImageFullPath => "Path/To/Test/Image/issue750-exif-tranform.jpg";

    [GlobalSetup]
    public void Setup()
    {
        this.sourceBytes = File.ReadAllBytes(this.TestImageFullPath);
        this.destBytes = new byte[this.sourceBytes.Length * 2];
    }

    [Benchmark]
    public void Series()
    {
        using var image = Image.Load(this.configuration, this.sourceBytes, new JpegDecoder { IgnoreMetadata = true });
        using var destStream = new MemoryStream(this.destBytes);

        image.Mutate(c => c.Resize(new ResizeOptions()
        {
            Mode = ResizeMode.Max,
            Size = new Size(800, 600)
        }));
        image.SaveAsJpeg(destStream);
        destStream.Seek(0, SeekOrigin.Begin);

        image.Mutate(c => c.Resize(new ResizeOptions()
        {
            Mode = ResizeMode.Max,
            Size = new Size(400, 250)
        }));
        image.SaveAsJpeg(destStream);
        destStream.Seek(0, SeekOrigin.Begin);

        image.Mutate(c => c.Resize(new ResizeOptions()
        {
            Mode = ResizeMode.Crop,
            Size = new Size(120, 120)
        }));
        image.SaveAsJpeg(destStream);
    }
}

And the reported average execution time on an Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R) CPU with .NET Core 3.1 is around 280 ms.

Any chance you can try this with your biggest test image on .NET Core 3.1?

Note: I guess it's unlikely to be the cause of your perf problems, but you may want to make sure that metadata is ignored during thumbnail generation (new JpegDecoder { IgnoreMetadata = true }),.

[Webreaper]

Yeah, it does seem weird. I will try and give it a go - might be able to find time to do it later.

For reference, I'm running my code in VS for Mac on a 2020 MacBook Air i5 1.1 GHz Quad-Core Intel Core i5.

[Webreaper]

I added a quick GetTickCount and the load takes an average of 1006ms for 200 images,
and the thumbnail gen (for 3 thumbs) takes 657ms (average for 200 images).
The AutoOrient call takes around 150ms.

By the way, just for context/comparison, running the same process on the same Macbook (also with Turbo disabled - so the same conditionas as the above) using SkiaSharp, the average load for 100 images is 335ms, and the average time to generate the 3 thumbnails is 135ms. Orienting the images takes around 9ms. So basically loading takes 3x as long with ImageSharp, and generating the thumbnails takes 4x as long.

[Webreaper]

Okay, so, running this with .Net 3.1 and your test image, it takes 918ms. Bear in mind that for reasons too painful to go into, I have turbo-boost disabled on my MacBook's (which is a 1.1Ghz quad-core i5). If I enable turbo boost, it drops to 442ms.

I ran this for one of my larger images. Mean time was 693ms (turbo enabled), and 1331ms with turbo disabled - which lines up approximately with the performance I've been seeing previously. So I guess part of this is just that I have a slow laptop/dev machine. ;)

These speeds are also consistent relative to the performance you see with your i7...

[antonfirsov]

with turbo disabled

Ok this explains everything, thanks for taking the time to rebenchmark! I was aware that we are relatively slower compared to Skia or even System.Drawing, but I was surprised by the absolute numbers you got, since it's not what we typically see.

So basically loading takes 3x as long with ImageSharp, and generating the thumbnails takes 4x as long.

Do you experience this by utilizing SKBitmap.Decode(codec, nearest) for the first thumbnail?

---

To put the perf topic into a historical context: We started with a perf gap about 10x bigger a few years ago, the first increments were relatively easy, but the closer we get to the theoretical maximum, the harder/more is the work to be done to improve. We hope that #1242 and #1121 will bring significant speedup here, probably bringing us closer to 2x gap compared to Skia. With some luck this should be all done in a couple of months.

On long term plans:
I really hope we can make the "decode to target size" feature to happen, but it's not everything. The total amount of work we theoretically can do is crazy in terms of time. Closing the performance gap entirely really needs people working on the library in full time, which is not the case unfortunately :( Sidenote: the more funding the project gets from selling support licenses, the more we can achieve in long term.

[Webreaper]

Do you experience this by utilizing SKBitmap.Decode(codec, nearest) for the first thumbnail?

Yes, exactly. My 'load' code is as follows:

        SKCodec codec = SKCodec.Create(source.FullName);
        SKImageInfo info = codec.Info;

        SKSize supportedScale = codec.GetScaledDimensions((float)desiredWidth / info.Width);
        SKImageInfo nearest = new SKImageInfo(supportedScale.Width, supportedScale.Height);
        SKBitmap bmp = SKBitmap.Decode(codec, nearest);

This was what made me think that if it was possible to do the 'nearest' decode within ImageSharp, you might be able to get a big performance win (although the disclaimer is that I know zero about JPEG decoding, so have no idea of the mechanics of this). As you said earlier, though, it may be that the efficiencies are memory-based rather than CPU-based.

I appreciate that what you're trying to do is hard, and that comparing to something like Skia is somewhat unfair, since writing performant low-level C++ is harder, but is always going to be faster if done well. If you can get close to 2x performance, that's a pretty significant achievement - particularly as you're doing it mostly in spare time. For my project, I'm going to keep the ImageSharp stuff in the codebase triggered by a config switch; even if SkiaSharp is faster, there may be platforms where it doesn't work or the native libs aren't available, so I want people to have the option to use the completely native code to ensure it'll work everywhere (i.e., slightly slower is better than non-functional).

Another interesting point - I also tried shelling out to GraphicsMagick to create the thumbnails, and it was significantly faster than even SkiaSharp. But in the end it proved too complex and unreliable.

[antonfirsov]

I'm going to keep the ImageSharp stuff in the codebase triggered by a config switch; even if SkiaSharp is faster, there may be platforms where it doesn't work or the native libs aren't available

That's a wise move 👍 (But OFC I'm opinionated here 😄 )

A sidenote that might be interesting or uninteresting for you, depending on your use-case:

If you are dealing with a content-management like scenario, where thumbnails are being consumed with much higher frequency, than new images uploaded to the system, you can implement caching of thumbnails. This will ideally outperform any decoder/resizer by several orders of magnitude. ImageSharp.Web is here to provide this feature out of the box for ASP.NET Core.

[Webreaper]

Thanks for the feedback. The app I'm writing is basically a sort of Blazor Server Lightroom, for extremely large collections of photos. Caching doesn't help; I'm actually already doing that - I have a background process that batch-creates all of the thumbnails and writes them to disk, so that all photos have all thumbnails available permanently. In my photo library (around 500k images/3TB of photos) it takes around 3-4 days to do them all with SkiaSharp (6-7 with ImageSharp) but then it's done, and they're only regenerated if an image changes.

I have thought about making the thumbnail generation real-time (rather than batch-converting up-front) but if somebody's scrolling through a list of 500 images, then generating the thumbs is going to take 2-3 minutes, which isn't really acceptable. But it's something I might consider (somebody who has a beefy CPU might prefer to take that option over using a couple of hundred MB of space on disk for the thumbs....). It's on my list....

[Webreaper]

Just flagging up the fact that this PR is going to solve the issue raised in this discussion: #2076 - wooooohooo! 👍