Support conversion to and from Tensors

[ynse01]

The @corefx team has created the System.Numerics.Tensors namespace.

Tensors are basically as n-dimensional arrays and are extensively used in Neural Networks.

A significant portion of Neural Networks has images as its input. I expect these datatypes to gain more traction quite quickly, given all the attention to Machine Learning in the media. Microsoft started off, by using these types in their own ONNX runtime library.

Would it be an good addition to ImageSharp to be able to convert to and from these Tensor types?

If so, would you like to add this to the Core library, or go with a seperate package, like ImageSharp.Textures?

I'll be happy to author or contribute either way.

[tannergooding]

The corefx team has created the System.Numerics.Tensors namespace.

While the ImageSharp maintainers have the final say here, this namespace is experimental and not stable. It is not something that should be used or consumed from stable libraries and we are very likely to change the surface, potentially substantially, before System.Numerics.Tensors actually ships non-experimentally.

[ynse01]

@tannergooding You are correct. However their NuGet package reached RTM level. So, as release (and stability) of their code is approaching, I think we can start discussing if Tensors are a useful addition to ImageSharp.

I agree with you, we should wait with the implementation until @corefx releases a final NuGet package.

[tannergooding]

So, as release (and stability) of their code is approaching

I am the owner and maintainer of System.Numerics.Tensors on the BCL team (https://github.com/dotnet/runtime/blob/main/docs/area-owners.md). The package is none of these things and the versioning is just a bi-product of it living in dotnet/runtime; the same as all experimental packages.

System.Numerics.Tensors is likely a .NET 8 feature at the earliest; and that's only if we end up taking a good solid look at the API surface and ensuring that it integrates properly with the "generic math" feature that previewed in .NET 6 and is planned to ship in .NET 7. I do not expect we will have time to do the in depth work required to improve the API surface in .NET 7 and if we did, it would likely ship as an updated preview/experimental surface.

[ynse01]

OK, wasn't aware of all that!

I gues you guys are still open for API related feedback then? I'll play around with the code some more and will file some issues on your Github page, if you like.

Btw, this also answers my original question: System.Numerics.Tensors is not a likely addition to ImageSharp any time soon due to its unstable API.

[skywalkerisnull]

@ynse01 We are in a similar situation as you, we are needing to move from an RGB image to a tensor, and to do it efficiently in dotnet has been a struggle. On our V100 we can infer a batch of 500 image tiles, in 200ms, but it takes just over 11 seconds to build that batch. Below is what we are are using to convert the RGB to the tensor (similar to https://github.com/microsoft/onnxruntime-csharp-cv-template/blob/af9061afe3077b2b7b137bf2d280541ecc316c4d/utils/ImageHelper.cs )

for (var j = 0; j < numberBatches; j++) {
  inputDimension[0] = batchSizes[j];
  Tensor<float> input = new DenseTensor<float>(inputDimension);

  for (var i = 0; i < batchSizes[j]; i++) {
    var image = tileList[tileIndex].Image;
    image.ProcessPixelRows(pixelAccessor => {

      for (var y = 0; y < image.Height; y++) {
        var rowSpan = pixelAccessor.GetRowSpan(y);

        for (int x = 0; x < image.Width; x++) {
          input[i, 0, y, x] = rowSpan[x].R;
          input[i, 1, y, x] = rowSpan[x].G;
          input[i, 2, y, x] = rowSpan[x].B;
        }
      }
    });
    tileList[tileIndex].Image = null;
    tileIndex++;
  }
  var tempList = new List<NamedOnnxValue>(){
      NamedOnnxValue.CreateFromTensor($"image", input)};
  batchList.Add(tempList);
}

Did you have any concepts or ideas on how to translate to the tensor faster? Or maybe there is a way to access the RGB array directly using a byte array? something similar to this: https://stackoverflow.com/questions/62470779/efficient-bitmap-to-onnxruntime-tensor-in-c-sharp

[antonfirsov]

@skywalkerisnull I think there are two bottlenecks in your code.

First is byte -> float conversion. You can improve this significantly by using ProcessPixelRowsAsVector4.

The other one is rooted in the way the code is copying the data:

input[i, 0, y, x] = r;
input[i, 1, y, x] = g;
input[i, 2, y, x] = b;

Assuming the tensors buffer is layed out from the most significant dimension to least significant one (which I don't know for sure), the subsequent index operators 0, 1 and 2, are accessing very distant memory areas, leading to frequent cache misses. I would try to change the tensor's indexing logic so you can do the following instead:

input[i, y, x, 0] = r;
input[i, y, x, 1] = g;
input[i, y, x, 2] = b;

I wish there was a way to make it even faster by just copying a Span<float> to DenseTensor<T>.Buffer, but that would require us to convert the image to a Vector3 buffer, which is not possible with the current API.

[skywalkerisnull]

Hey @antonfirsov we will give the ProcessPixelRowsAsVector4 ago.

Unfortunately, we cannot change the shape of the tensor as this is the standard layout, [Batch, Channel, Height, Width] the only way that we could really do that would be if the system.numeric.tensors supported a transpose operation (which I do not believe it currently has). Or we would need to implement the transpose ourselves. There is some more details on the NCHW vs NHWC here: https://github.com/microsoft/onnxruntime-openenclave/blob/openenclave-public/docs/ONNX_Runtime_Perf_Tuning.md#my-converted-tensorflow-model-is-slow---why

[antonfirsov]

Or we would need to implement the transpose ourselves.

Without SIMD it's going to suffer from the same issues as your current RGB code. I think runtime is the right place for such optimizations, so I recommend to raise a feature request in dotnet/runtime. You can mention NCHW <-> NHWC conversion as a use-case.

---

I realized that there is a third optimization technique that can be even more significant than the two above. The indexing operator seems to have a very high overhead:

• https://github.com/dotnet/runtime/blob/main/src/libraries/System.Numerics.Tensors/src/System/Numerics/Tensors/Tensor.cs#L654
• https://github.com/dotnet/runtime/blob/eeff29824557cbc3873c2d331d042a889310bac0/src/libraries/System.Numerics.Tensors/src/System/Numerics/Tensors/ArrayUtilities.cs#L126

Avoid it like the plague and slice down DenseTensor<T>.Buffer.Span manually as you traverse the dimensions. We are exposing GetRowSpan(y) to have API support for this technique in ImageSharp.

[skywalkerisnull]

In case anyone else stumbles across this. I was able to eventually figure out a way with assistance from Microsoft. Below is a link to the model that will work significantly faster. We went from approx. 11.2 seconds per batch, to approx. 0.3 seconds per batch

https://github.com/skywalkerisnull/onnxruntime-csharp-cv-template/blob/82bbcb8c333c82eda488b69f69b735f073a1628a/utils/ImageHelper.cs#L55

[ynse01]

Great result @skywalkerisnull !

Looking at the code, I still see a number of performance opportunities:

1. In the y loop, you call GetIndex 3 times. The same can be accomplished by replacing those calls with index += strides[1]
2. In fact, the same can be applied to the other looping levels, by adding the correct strides in each loop iteration. This would eliminate mdIndex and GetIndex()
3. In the inner most loop, you divide twice. The division with 255f can be eliminated by pre-multiplying the mean and stddev values with 255 outside of the loops
4. The spanR, spanG and spanB calculations can be further speed up by using Numerics
5. You can replace tileIndex with i, their values are always the same

[skywalkerisnull]

@ynse01 Done! https://github.com/skywalkerisnull/onnxruntime-csharp-cv-template/blob/bb1454a51a722e293a918d2b5a25abda864f9f74/utils/ImageHelper.cs#L62

I will have to test these changes to see how much of an impact they have on the production system. In our production systems, we don't actually normalise in the C# code, we moved that a long time ago into the DNN graph (if we do it at all).

When you say numerics, I am guessing you are referring to making multiple Vector3 and doing the math across the vectors and then extract the value of the vector for each span?

[ynse01]

You're getting closer, let's discuss the details:

1. You nailed this one !
2. Line 97, you can replace with index += strides[2]. This works as it will be called exactly y times in the loop.
3. What I meant was slightly different. Goal is to take all multiplications with 255 outside of the loop, where they are execute less times (only once). Outside of the loop is around line 75, create variables meanNorm0 = norm[0] * 255f; and stddevNorm0 = stddev[0] * 255f;. Do the same for index 1 and 2. Line 112 then becomes: spanR[x] = (rowSpan[x].R - meanNorm0) / stdevNorm0;. Line 113 and 114 are similar.
4. For inspiration into a Numerics version of this code, look at ImageSharp's YCbCr conversion code.
5. This one is already OK.

[skywalkerisnull]

Hey @ynse01,

2. Line 97, you can replace with index += strides[2]. This works as it will be called exactly y times in the loop.

I don't think that line 97 can be changed to just increment by the stride[2] as we are already jumping around quiet a lot in the lines below. If I change that line to index += strides[2] we get an out of range exception on the slice, the same happens if I try to move index += strides[2] to the end of the loop to mitigate that it actually needs to start at index position of 0. But looking at it closer, I did have an extra index += strides[1]; that was not required at line 107

3. What I meant was slightly different. Goal is to take all multiplications with 255 outside of the loop, where they are execute less times (only once). Outside of the loop is around line 75, create variables meanNorm0 = norm[0] * 255f; and stddevNorm0 = stddev[0] * 255f;. Do the same for index 1 and 2. Line 112 then becomes: spanR[x] = (rowSpan[x].R - meanNorm0) / stdevNorm0;. Line 113 and 114 are similar.

This now makes more sense. I spent about 30 minutes in front of my whiteboard trying to do the algebra to factor out so that it was just the form of spanR[x] = rowSpan[x].R * normFactorR; 😂 But further to this, if we do something like this out side the loop: var stdNormR = 1 / (255f * stddev[0]); and then multiply that instead of dividing, would that be faster? My testing on my laptop is inconclusive.

4. For inspiration into a Numerics version of this code, look at ImageSharp's YCbCr conversion code.

I will have to have a deeper look into this

[br3aker]

But further to this, if we do something like this out side the loop: var stdNormR = 1 / (255f * stddev[0]); and then multiply that instead of dividing, would that be faster? My testing on my laptop is inconclusive.

Yes, it will be faster simply because instead of

spanR[x] = (rowSpan[x].R / (255f * stddev[0])) - normR;
spanG[x] = (rowSpan[x].G / (255f * stddev[1])) - normG;
spanB[x] = (rowSpan[x].B / (255f * stddev[2])) - normB;

you'll get

spanR[x] = rowSpan[x].R / premul_stddev[0] - normR;
spanG[x] = rowSpan[x].G / premul_stddev[1] - normG;
spanB[x] = rowSpan[x].B / premul_stddev[2] - normB;

Which is already a big deal because you now have less operations to execute. Multiplying by a reciprocal value:

spanR[x] = rowSpan[x].R * reci_premul_stddev[0] - normR;
spanG[x] = rowSpan[x].G * reci_premul_stddev[1] - normG;
spanB[x] = rowSpan[x].B * reci_premul_stddev[2] - normB;

Would yield even more performance because multiplication is usually a lot faster than division.

This code can be 'vectorized' via sse/avx on x86 or some vector intrinsic stuff on arm/etc. For example. sse would calculate these values in batches of 4, avx with batches of 8 floats per cpu operation.

Finally, even vector implementation can be accelerated because you have a very fortunate pattern: multiplication followed by an addition/substraction, this can be exploited via x86 extension - FMA (fused-multiply-add). Fma operations allow you to do multiply-add chain for the price of a single multiply, i.e. addition is free.

To conclude: from 3 divisions, 3 multiplications and 3 substractions we got to only 3 multiplications with batches of 8 floats per. It's hard to guess exact performance boost from this but it'll be a lot faster than current scalar implementation :)

P.S.
To be super nerdy: you have 3 same operations in a row - that's where cpu pipelining kicks. This allows cpu to execute these operations faster than if you had single operation within loop.