ProjectiveTransformBuilder from CSS Matrix3d transform

[NiftyImages]

I am attempting to reproduce the matrix3d CSS transform in ImageSharp. I have to admit that I am out of my element here, but I was hoping that someone could point me in the right direction.

I am using MovableJS to do the warping on the web interface, and it results in a CSS Transform that looks like this:

transform: rotate(0deg) translate(0px, 0px) scaleX(1) scaleY(1) matrix3d(0.578352, -0.00235739, 0, -1.25508e-05, 0.0626585, 1.00757, 0, -0.00224878, 0, 0, 1, 0, -12.8798, -80.4789, 0, 0.999998)

Here is a screenshot of the interface for perspective (pun intended):

I get the values from matrix3d property and try to use a ProjectiveTransformBuilder as follows:

var matrix = new Matrix4x4(0.578352f, -0.00235739f, 0, -1.25508e-05f, 0.0626585f, 1.00757f, 0, -0.00224878f, 0, 0, 1, 0, -12.8798f, -80.4789f, 0, 0.999998f);

image.Mutate(t => {
    ProjectiveTransformBuilder builder = new ProjectiveTransformBuilder();
    builder.AppendMatrix(matrix);
    t.Transform(builder);
});

The result looks like this:

The matrix3d css transform is a 4x4 homogeneous matrix (documentation). I've been able to AffineTransformBuilder translations and rotation, but this is really throwing me for a loop.

Am I thinking about this correctly, is it not as simple as applying the same matrix to the image before drawing on the background?

I've been at this for a couple days now, any help would be greatly appreciated.

[JimBobSquarePants]

Hi @NiftyImages

I'm going to have to do some experimentation to figure out the issue here. Our matrix transforms are normally sound.

ImageSharp

var matrix = new Matrix4x4(
    0.578352F, -0.00235739F, 0, -1.25508e-05F,
    0.0626585F, 1.00757F, 0, -0.00224878F,
    0, 0, 1, 0,
    -12.8798F, -80.4789F, 0, 0.999998F);

using var image = new Image<Rgba32>(200, 200, Color.Black);

image.Mutate(t =>
{
    ProjectiveTransformBuilder builder = new ProjectiveTransformBuilder();
    builder.AppendMatrix(matrix);
    t.Transform(builder);
});

image.Save(Path.Combine(outPath, "1744.png"));

I experimented with Skia also and it also does not appear render that matrix as expected. What is the size of the input shape?

using (var bitmap = new SKBitmap(200, 200))
{
    // How do you fill a skia image?
    for (int y = 0; y < bitmap.Height; y++)
    {
        for (int x = 0; x < bitmap.Width; x++)
        {
            bitmap.SetPixel(x, y, new SKColor(0, 0, 0, 255));
        }
    }

    SKMatrix44 matrix44 = SKMatrix44.FromColumnMajor(new[]
    {
        0.578352F, -0.00235739F, 0, -1.25508e-05F,
        0.0626585F, 1.00757F, 0, -0.00224878F,
        0, 0, 1, 0,
        -12.8798F, -80.4789F, 0, 0.999998F
    });


    using var surface = SKSurface.Create(new SKImageInfo(200, 200, bitmap.ColorType, bitmap.AlphaType));
    using var paint = new SKPaint() { FilterQuality = SKFilterQuality.High };

    var canvas = surface.Canvas;
    canvas.SetMatrix(matrix44.Matrix);
    canvas.Clear();
    canvas.DrawBitmap(bitmap, 0, 0, paint);
    canvas.Flush();

    using var output = File.OpenWrite(Path.Combine(outPath, "1744-skia.png"));
    surface.Snapshot()
        .Encode(SKEncodedImageFormat.Png, 75)
        .SaveTo(output);
}

[NiftyImages]

Thanks for having a look.
The background image is 800x534 and the chalkboard is 600x375.

I hate to bother you guys with these questions, but like I said....I've tried a bunch of different combinations.
I load the graph paper (background image), then apply the transformation to to the chalkboard before.

Not sure if I should do things in a different order.
Here is a simplified copy of my code:

`
using (Image image = Image.Load ("grids.jpg")) {

var imageBytes = await InternalHelper.GetChalkboardImageBytes ();

using (var chalkboard = Image.Load (imageBytes)) {

    var matrix = new Matrix4x4 (0.578352f, -0.00235739f, 0, -1.25508e-05f, 0.0626585f, 1.00757f, 0, -0.00224878f, 0, 0, 1, 0, -12.8798f, -80.4789f, 0, 0.999998f);

    chalkboard.Mutate (t => {
        ProjectiveTransformBuilder builder = new ProjectiveTransformBuilder ();
        builder.AppendMatrix (matrix.Value);
        t.Transform (builder);
    });

    image.Mutate (x => {
        x.DrawImage (chalkboard, new Point (el.X, el.Y), 1);
    });

}

using (var mem = new MemoryStream ()) {
    await image.SaveAsPngAsync (mem);
    return mem.ToArray ();
}

}`

Background Image:

Chalkboard I'm trying to transform:

[JimBobSquarePants]

I can explain why the output is cut off at least.

It contains a translation component at M41, M42 which shifts the entire object up and left. Without that component you end up with the following.

Normally for transforms like what you are after I would recommend using tapering.

using var board = Image.Load<Rgba32>(Path.Combine(inPath, "1744-fg.png")); 
using var grid = Image.Load<Rgba32>(Path.Combine(inPath, "1744-bg.jpg"));
board.Mutate(t =>
{
    ProjectiveTransformBuilder builder = new ProjectiveTransformBuilder();
    builder.AppendTaper(Taperside.Top, Tapercorner.Both, .4F);
    t.Transform(builder);
});

int left = (grid.Width - board.Width) / 2;
grid.Mutate(x => x.DrawImage(board, new Point(left, 82), 1));
grid.Save(Path.Combine(outPath, "1744.png"));

[JimBobSquarePants]

Our Taper Transform implementation is based of example code for Skia given here
https://docs.microsoft.com/en-us/xamarin/xamarin-forms/user-interface/graphics/skiasharp/transforms/non-affine