Overlapping blocks: here or a new package?

[timholy]

@kescobo was asking in the #image-processing slack about assembling a bunch of individual, overlapping images into a single larger image while maintaining the ability to access each individual underlying array. The translations needed to align the images in their regions of overlap is done by external tools, but once you know what those translations are, this seems quite a lot like "mortar + overlaps". We wondered whether it would be better to add this to BlockArrays.jl, or whether it would complicate things here to the point that it's better done as an external package. Just wondering what the maintainers of this package would prefer.

@kescobo, I forgot to ask: are all the transformations pure translations? If there are rotations/etc, presumably that might be important to know at the outset.

[jishnub]

I'd say that it's best to start as a separate package, as there's not enough bandwidth to actively maintain this use case over here. If, at some point, the package stabilizes, and the maintainers feel that the two packages may be merged, we may consider that.

[dlfivefifty]

Can you express this in terms of linear algebra?

Let me see if I can understand:

1. Images are Matrix{<:Color}
2. You want to support BlockMatrix{<:Color, Matrix{<:Matrix{<:Color}}}. This will probably work to some extent without any modifications provided that all the images have compatible sizes (Though perhaps we want better output to show the image with the block structure added on top)
3. But you also want to support images with incompatible sizes? In which case you need to stretch them or something?

If I understand right, maybe something like a TranslationMatrix would help? Then the type could be BlockMatrix{<:Color, <:Matrix{<:TranslationMatrix{<:Color}}}. But then this could ive in another package...

[timholy]

I do think it needs a new type. The key issue is one wants something like this:

A = [ 1  2  3  4;
      5  6  7  8;
      9 10 11 12]
B =       [ 7  8 13 14;
           11 12 15 16]

A and B agree on the block

 7  8
11 12

and thus B should be translated by (1, 2) so that it overlaps A in that region. Note the redundancy between A and B for those coordinates; I'd propose that for indexing, the block with lower translation indices "wins." The call to construct such an array might be C = mortar([A, B]; translate=[(0,0), (1,2)]) with "virtual" result

[ 1  2  3  4 missing missing;
  5  6  7  8 13 14;
  9 10 11 12 15 16]

In real imaging contexts, there will be noise, so the agreement will not be numerically exact.

[timholy]

I'll leave this open for a little bit longer in case there is additional discussion, but at this moment it sounds like putting this in a separate package makes the most sense. Don't hesitate to correct me or expand any of this discussion, but preliminarily I want to thank both of you for the very rapid feedback!

[kescobo]

On my phone at the moment - will add some more context when I'm at a computer just to complete the record, but this all makes sense to me 👍

[kescobo]

You want to support BlockMatrix{<:Color, Matrix{<:Matrix{<:Color}}}. This will probably work to some extent without any modifications provided that all the images have compatible sizes (Though perhaps we want better output to show the image with the block structure added on top)

Yeah - right now the display (or show?) method errors:

EDIT: this doesn't seem to happen in a fresh session, I must have screwed something else up 🤔

Previously erroring code

julia> img1 = rand(RGB, 4,5);

julia> img2 = rand(RGB, 4,5);

julia> mortar([[img1] [img2]])
Error showing value of type BlockMatrix{RGB{Float64}, Matrix{Matrix{RGB{Float64}}}, Tuple{BlockedOneTo{Int64, Vector{Int64}}, BlockedOneTo{Int64, Vector{Int64}}}}:
ERROR: MethodError: Cannot `convert` an object of type RGB{Float64} to an object of type Float64

The main thing I wanted was to have a structure that allows me to retain the original images, and have a view-like super structure. It seems like BlockArrays already supports this - I can even provide views into the original as components. Eg, the following seems to work:

julia> img2v = @view img2[:, 2:end];

julia> blocks = mortar(reshape([img1, img2v], 1,2));

julia>  blocks[Block(1,2)]

julia> img2[3,:] .= RGB(0,0,0); # the original image

julia> blocks[Block(1,2)]

So aside from the display, getting fixed (which I think would happen here? let me know if I should open a separate issue), I think the BlockOverlaps.jl or whatever would just need to wrap the existing types, and have ways to store originals + views.

In this example, I'd like to be able to do something like blocks[Block(1,2)][:, 1] and get the first column of the original, rather than the first column of the view. But I think this should be straightforward. And I have to say, the fact that I can already do this is kind of amazing:

[kescobo]

The error I noted above is an error when SixelTerm is also loaded (xref eschnett/SixelTerm.jl#13)

I'm not sure where the error originates