Plot Images

[dcnieho]

It would be super cool if you are able to draw images positioned at a certain 3D position/orientation.

According to some googling its possible in matlab like this:
https://se.mathworks.com/matlabcentral/answers/300826-how-do-i-insert-an-image-in-my-2-d-and-3-d-plots-in-matlab-8-2-r2013b
which produces:

If i mess with a corner point this still works fine:

Being able to texture surfaces would be super helpful for applications, e.g., where you show some 3D reconstruction result.

I guess it is possible using custom rendering, adapting your example there. But then it would be nice if the library would support a way to z-sort for the current frame so that i can draw back to front. That would entail submitting geometry and then being able to iterate over the plot objects and having them returned in the correct order so occlusions would be correct.

[brenocq]

I love the idea, thanks for the contribution! This should be possible to implement and it sounds very useful, I'll take a look into it later this week! :)

I guess it is possible using custom rendering, adapting your example there. But then it would be nice if the library would support a way to z-sort for the current frame so that i can draw back to front.

ImPlot3D implements a "triangle" z-sort internally to make sure triangles are rendered in the correct order. It is not ideal when triangles overlap, but it works nice most of the time.

[dcnieho]

Cool, looking forward to it! If you implement it directly, some kind of access to your internal z-sorting would not be needed (though potentially still useful for other applications of course, I'd call it pretty low priority at best).

[brenocq]

Hey @dcnieho! Just letting you know I started working on the ImPlot3D::PlotImage API in the branch plot-image. Below are some of the initial results

Peek.2025-03-13.09-15.mp4

When the image corners are too distorted, it doesn't look that great because of the way ImGui renders triangles. As far as I know, we would need a fragment shader to render quads with homography transformation to render distorted quads correctly (maybe future work? 👀).

There’s still a bit of work left to support custom textures, but it’s getting there!

The API currently looks like this:

IMPLOT3D_API void PlotImage(const char* label_id, ImTextureID user_texture_id, const ImPlot3DPoint& p0, const ImPlot3DPoint& p1,
                            const ImPlot3DPoint& p2, const ImPlot3DPoint& p3, const ImVec2& uv0 = ImVec2(0, 1), const ImVec2& uv1 = ImVec2(1, 1),
                            const ImVec2& uv2 = ImVec2(1, 0), const ImVec2& uv3 = ImVec2(0, 0), const ImVec4& tint_col = ImVec4(1, 1, 1, 1),
                            ImPlot3DImageFlags flags = 0);

I’m also working on an additional API for simpler image plotting, where users can define an image as a square/rectangle without manually specifying each corner’s position and UVs.

Let me know if you have any feedback!

[dcnieho]

@brenocq: This looks super nice, thanks! does it interact correctly with other plots like lines and surfaces in terms of occlusion and such?
I am very happy to be able to use this :)

[brenocq]

Hello! The depth testing currently happens per triangle, so it works reasonably well when the triangles being rendered do not intersect. In the video below, the image is rendered as two large triangles, and each line plot as many small triangles (each line segment is rendered as two triangles), so it works well most of the time :)

Peek.2025-03-23.09-23.mp4

In the future, I hope to have depth testing per pixel, which should resolve most depth testing issues.

[dcnieho]

That looks really nice, thanks again!

[brenocq]

I just finished implementing support for custom textures in ImDrawList3D. Below is an example of rendering three different images in the same plot:

Peek.2025-03-23.16-45.mp4

To support multiple textures within the ImGui rendering pipeline, each unique texture requires a separate draw call. At the same time, we still need to ensure triangles are rendered in the correct order based on Z depth. I tried to keep overhead minimal -- so if no PlotImage calls are made, the plot remains as efficient as before (rendered with a single draw call).

However, when custom images are used, we may need multiple draw calls. The image below illustrates how draw calls are generated from an ImDrawList3D containing 5 textured triangles:

In this case, ImDrawList3D contains 5 triangles -- each with a different Z depth and TextureID (triangles are color-coded by the TextureID in the image). After sorting the triangles by depth, draw calls are split whenever the texture changes, resulting in one draw call per contiguous group of triangles sharing the same texture.

To implement this, I added a _TextureBuffer to ImDrawList3D, and updated ImDrawList3D::SortedMoveToImGuiDrawList to emit multiple draw calls when needed.

I'm going to work on simplifying the PlotImage API now. I think it will be ready for the PR after that :)

[brenocq]

Done! The following two APIs were implemented to plot images.

// Plots a rectangular image in 3D defined by its center and two direction vectors (axes).
// #center is the center of the rectangle in plot coordinates.
// #axis_u and #axis_v define the local axes and half-extents of the rectangle in 3D space.
// The rectangle is formed by moving from the center along ±axis_u and ±axis_v.
// #uv0 and #uv1 define the texture mapping.
// #tint_col can be used to tint the image.
IMPLOT3D_API void PlotImage(const char* label_id, ImTextureID user_texture_id, const ImPlot3DPoint& center, const ImPlot3DPoint& axis_u,
                            const ImPlot3DPoint& axis_v, const ImVec2& uv0 = ImVec2(0, 0), const ImVec2& uv1 = ImVec2(1, 1),
                            const ImVec4& tint_col = ImVec4(1, 1, 1, 1), ImPlot3DImageFlags flags = 0);

// Plots an image using four arbitrary 3D points that define a quad in space.
// Each corner (p0 to p3) corresponds to a corner in the image, and #uv0 to #uv3 are the texture coordinates for each.
// This overload allows full control over orientation, shape, and distortion.
// Note: The quad is internally split into two triangles, so non-rectangular quads may produce rendering artifacts
// since distortion is interpolated per triangle rather than over the full quad.
IMPLOT3D_API void PlotImage(const char* label_id, ImTextureID user_texture_id, const ImPlot3DPoint& p0, const ImPlot3DPoint& p1,
                            const ImPlot3DPoint& p2, const ImPlot3DPoint& p3, const ImVec2& uv0 = ImVec2(0, 0), const ImVec2& uv1 = ImVec2(1, 0),
                            const ImVec2& uv2 = ImVec2(1, 1), const ImVec2& uv3 = ImVec2(0, 1), const ImVec4& tint_col = ImVec4(1, 1, 1, 1),
                            ImPlot3DImageFlags flags = 0);

Below is the final demo

Peek.2025-03-23.18-41.mp4

[brenocq]

Hi @dcnieho! #79 was finally merged into main! Let me know if you have any problems with it! I'll closing this task now.

[dcnieho]

Super, thanks so much! I look forward to using it.