Is there a way to avoid overflows with mutations such as DetectEdges?

[ahardin]

Note, I am pretty new to image processing and my knowledge of the math and algorithms mentioned below is somewhat superficial, but I believe I have the concept right. Correct me if I'm wrong/misguided.

As I understand, calling Mutate on an image replaces the Image's pixel values with the result of the mutation.

In the case of something like the Laplacian3x3 edge detection kernel, pixel values are very likely to undergo a multiplication operation whose product exceeds the MaxValue of the underlying datatype of most (all?) PixelFormats.

For example, if you have an L8 image and a pixel whose value is 50, the convolution kernel may multiply it by 8 (oversimplifying of course). The result is of course 400, but since it gets cast(?) back into a byte, it winds up being 144. Similarly, the edge detection kernels often use negative numbers, which leads to the same type of behavior.

By way of comparison, in OpenCV, when you run filters like Laplacian or Sobel, you can specify the depth of the resulting image in order to avoid this behavior and therefore retain the true results of the filter/mutation (see comment about the ddepth parameter on this page: https://docs.opencv.org/4.6.0/d5/db5/tutorial_laplace_operator.html).

Interestingly, the resulting edge-detected image appears to be identical (or nearly so) in both approaches (ImageSharp allowing the overflow, and OpenCV allocating a larger datatype). I'm curious if somebody could explain why that is, but it's sort of beside the point of my main discussion/question.

The problem for me is I would like to retain the true results of the Laplacian filter. Is this possible?

A little more background...

I'm experimenting with using Laplacian edge detection to determine if an image is blurry (see: https://pyimagesearch.com/2015/09/07/blur-detection-with-opencv/).

The general idea is:

1. Convert image to grayscale
2. Apply Laplacian 3x3 filter
3. Calculate variance of resulting pixels
4. Higher variance = more clear, lower variance = more blurry

Since I'm basically trying to measure the differences between all of the pixel values, I think it's important that the actual numbers calculated by the kernel be retained, even though they are outside the bounds of the original PixelFormat.

Am I misunderstanding something, and/or is there a different API I should be using? I did look into Cloning the image to a larger PixelFormat like Short4, but that of course happens before any mutations and the pixel values appear to be scaled up, leading to the same behavior.

Here is some code and a test image:

Test image:

Code to load the bananas, convert to grayscale, apply Laplacian3x3, save image and output some stats based on the resulting pixels.

using var image = await Image.LoadAsync<Rgb24>("banana.png");

image.Mutate(z => z.Grayscale()); // Alternatively I could Load the image with L8, but the result is substantively identical

// make a kernel the same as OpenCV's default
var kernel = new EdgeDetectorKernel(new DenseMatrix<float>(new float[3, 3] { { 0, 1, 0 }, { 1, -4, 1 }, { 0, 1, 0 } }));

image.Mutate(x => x.DetectEdges(kernel, false));

await image.SaveAsPngAsync("banana_lap.png");

// np is NumSharp
var arr = np.array(new float[image.Height, image.Width]);

// stick all the images pixels in a NumSharp array so we can easily do some stat calculations
image.ProcessPixelRows(accessor =>
{
    for (int y = 0; y < accessor.Height; y++)
    {
        var pixelRow = accessor.GetRowSpan(y);

        for (int x = 0; x < pixelRow.Length; x++)
        {
            ref Rgb24 pixel = ref pixelRow[x];
            arr[y, x] = pixel.R;
        }
    }
});

Console.WriteLine($" min: {np.min(arr)} max: {np.max(arr)} sum: {np.sum(arr)} var: {np.var(arr)}");

Resulting image and console output:

Here's the same operation in OpenCV.

import cv2 
import numpy as np
image = cv2.imread('banana.png')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
lap = cv2.Laplacian(gray, cv2.CV_64F) # note the CV_64F specifying the depth of the output image
print(f" min: {np.min(lap)} max: {np.max(lap)} sum: {np.sum(lap)} var: {np.var(lap)}")
cv2.imwrite('banana_lap_cv.png', lap)

Resulting image and output:

As mentioned, the resulting images are the same, but the results of any sort of analysis on the underlying data is very different.

[antonfirsov]

I didn't go too deep into your examples, but wouldn't loading the image into a floating point pixel type RgbaVector solve your problem? Or you can just take a higher resolution integer format like Rgb48 (or better L16 if you work in grayscale anyways), and maybe scale down the values of your kernel if you are afraid of overflows.

[brianpopow]

Interestingly, the resulting edge-detected image appears to be identical (or nearly so) in both approaches (ImageSharp allowing the overflow, and OpenCV allocating a larger datatype). I'm curious if somebody could explain why that is, but it's sort of beside the point of my main discussion/question.

In your ImageSharp example you are using Rgb24 as a pixel type. The result will be scaled to fit the range of that format, so it will be between 0 and 255.
OpenCv will do the same when you store the data as a png image. That's why the results look (almost) the same.

You can verify this when you reload the image:

reload = cv2.imread('banana_lap_cv.png', cv2.IMREAD_GRAYSCALE)
print(f"reload min: {np.min(reload)} max: {np.max(reload)} sum: {np.sum(reload)} var: {np.var(reload)}")

You can also achieve the same with:

# converting to uint8
abs_dst = cv2.convertScaleAbs(lap)
print(f"abs lab min: {np.min(abs_dst)} max: {np.max(abs_dst)} sum: {np.sum(abs_dst)} var: {np.var(abs_dst)}")

Another way to achieve the same result is when you provide -1 to the ddepth parameter of cv2.Laplacian.
This will make sure the resulting image will be the same scale as the input.

Some more things I noticed:

• The default GrayScale conversion of ImageSharp and opencv is different. You can get the same result, if in ImageSharp with image.Mutate(z => z.Grayscale(GrayscaleMode.Bt601))
• The border is treated different as default in opencv then in ImageSharp. You can get the same result with lap = cv2.Laplacian(gray, -1, borderType = cv2.BORDER_REPLICATE)

Lastly I am not sure that RgbaVector will solve the problem mentioned above. It is floating point, but it's in the range of 0 to 1.
I dont think we have a pixel type which replicates CV_64F exactly.

[JimBobSquarePants]

Would be simple enough for someone to make externally if they know the upper and lower bounds of CV_64F

[antonfirsov]

It is floating point, but it's in the range of 0 to 1.

A pixel type that doesn't clamp or a new PixelConversionModifier might help with this. It would be necessary to normalize the image after the operation though. (Finding the maximum pixel value, scaling [0..max] to [0..1].)

ahardin note that you can implement your own pixel format.

[ahardin]

@brianpopow Thank you very much for the detailed reply! Very enlightening.

@antonfirsov Thank you for your replies as well. A custom pixel format without clamping behavior sounds like it might be the way to go. I'll give that a shot and report back here.

Thanks again!