Remove Transform hierarchy in UI

[djeedai]

This is inspired by the work of @IDEDARY from this Showcase thread on Discord. They wrote a UI which uses a novel path-based approach for UI positioning. One take away from the discussion was that the Parent and Transform hierarchy commonly used elsewhere actually get in the way when it comes to UI:

• To support multi-window, there's already a UiStack.
• Taffy, used for layouting, already has its own internal representation of the UI hierarchy, which duplicates the one Bevy uses.
• The Transform of a Node is not directly related to its actual position on screen. In fact, if you look at the code to calculate the Node's relative position used by the system updating the RelativeCursorPosition component, you will find that it's using the Window, the Node, the GlobalTransform, and the CalculatedClip. That conversion is very much non-trivial, and not exposed anywhere, although it's relatively common to have to know the absolute position of a node on screen for various operations (especially mouse cursor interaction, drag-and-drop, hovering, etc.). The value of the Transform appears completely random to users, unrelated to what they set with Style.
• Related to this: unlike all other uses, Transform on UI nodes is auto-generated instead of being user set. This is quite confusing.
• The Transform propagation to parents to calculate GlobalTransform is entirely redundant with the layouting of UI, which already involves calculating the position of nodes relative to their parents, and can calculate an absolute position at the same time.

The proposal here would be to accept that Transform is just not the right tool for the job of positioning UI nodes, and Parent not the right tool to build a UI node hierarchy, and stop using them in the UI definition. It's possible that we still want to convert UI node in the main World to something else in the render World that's Transform-based, because we probably want to keep GlobalTransform for rendering. But for users in the main World it's probably more confusing than anything at the minute.

[nicopap]

I designed cuicui_layout.

Transform in UI

cuicui_layout — I think — is a clear example of how a layouting algorithm would look like if it integrated fully with bevy's transform system. I was honestly shocked how well it worked. Since I never dabbed in layout algorithm design before, and it just worked after 3 days of work. And this works so smoothly and it's so easy to implement, mostly because it works on top of the bevy transform propagation system!

In any case, I think your proposition is incomplete. You say:

it's relatively common to have to know the absolute position of a node on screen for various operations

But you don't provide an alternative in a world without Transform in UI. How would you compare the position of UI components for — for example — gamepad movement? And mouse picking? You don't mention it.

You say:

unlike all other uses, Transform on UI nodes is auto-generated instead of being user set.

This is false. Just last week I answered a question in the bevy discord regarding animations. Animations control skinned mesh bone Transform, and if your system updates the Transform at the correct point in the frame, it will be overwritten. So there is already precedent. (note that this doesn't make it something we wish to do, but it exists)

This also uncovers an interesting link between animation and layouting. But I don't care to dig into this right now.

Finally. The issue we have here is not of using Transform in UI. But rather that we are managing nodes in the taffy transform space, and translating it into bevy's Transform space in real time. This impedance mismatch results in a bit of confusion, I agree on that. But fundamentally pointing out Transform is pointing out the victim and blaming them for the actions of the perpetrator.

Hierarchy in UI

I really like 1DEDARY's solution though. Currently, hierarchy in bevy suck. (I can already hear Yoggi & sanders say "but it's better with relations"). And a solution that enables abstracting the hierarchy into its own module would allow avoiding that.

But maybe this also is blaming the victim for the actions of the perpetrator. Is it because bevy_hierarchy sucks that it's difficult to interact with hierarchies, or is it because it's difficult to interact with hierarchies that bevy_hierarchy sucks?

Hierarchies in UI are useful beyond just layouting stuff. bevy_eventlistener demonstrates usage of bubbling, which depends on the hierarchy.

bevy-ui-navigation also uses hierarchies to mark interactive elements as part of a single menu, and navigation is bound within a single menu.

Now, I agree it's an absolute misery to spawn UI elements using bevy_ui today. But again, I think pointing at the hierarchy for this is pointing at the victim, and not the perpetrator.

Synthesis and tout le tra-la-la

What we learned:

• Transform is not relevant to bevy_ui, but only because bevy_ui's positioning is entirely delegated to taffy.
• hierarchy is not relevant to bevy_ui but very relevant to crates building on top of it. It might be hard to work with it today, but it's likely due to bevy's limitations.

[djeedai]

Thanks for the feedback and shared thoughts. I think we agree on a number of points, even if my initial post was incomplete indeed.

But you don't provide an alternative in a world without Transform in UI. How would you compare the position of UI components for — for example — gamepad movement? And mouse picking? You don't mention it.

I'm not saying that we should remove the concept of node position from the public API. Quite the opposite in fact. My recent pain points are precisely related to those examples like mouse picking, where mouse cursor position cannot be compared to Transform or GlobalTransform, which represent weird derived quantities not equal to the node's position (for example, top left corner, or even center). And there's no direct access to, say, the position of the node's top left corner on screen. So the proposal is to get rid of Transform in favor of some UI position component/field.

So there is already precedent. (note that this doesn't make it something we wish to do, but it exists)

I didn't know about Animation. But I think we agree that this mix of behaviors is bad anyway.

Finally. The issue we have here is not of using Transform in UI. But rather that we are managing nodes in the taffy transform space, and translating it into bevy's Transform space in real time. This impedance mismatch results in a bit of confusion, I agree on that. But fundamentally pointing out Transform is pointing out the victim and blaming them for the actions of the perpetrator.

Not exactly. The issue is more with the fact that the Transform doesn't represent anything useful from a UI perspective. And that's unrelated to taffy, but rather I think to the fact we draw the UI with sprites and so need the Transform for rendering.

Hierarchies in UI are useful beyond just layouting stuff. bevy_eventlistener demonstrates usage of bubbling, which depends on the hierarchy.

100% agree. Layouting was the obvious example, but mouse cursor position for picking or event bubbling are also prime examples.

Transform is not relevant to bevy_ui, but only because bevy_ui's positioning is entirely delegated to taffy.

I disagree on that. Transform is irrelevant because it represents a 3D transform in world units, whereas UI uses 2D Val which can be represented in various units. And it makes Transform change behavior, from being the source of truth in most non-UI entities, to being a calculated byproduct of the UI's own positioning system. It's the wrong representation for a UI position. And even if you want to use it for the final calculated pixel position of the node, this will conflict with the rendering position imposed by the sprite renderer, so those two will always fight to use Transform as more convenient for them. As for Parent, its main role is to mark hierarchy for GlobalTransform calculation. So it could be reused for UI, but that would probably just be confusing for users, and we might as well have a different component for UI hierarchy.

[IDEDARY]

Hello guys. Even though Bevy_Lunex is not finished, I decided to open source my code ahead of time due to the interest it created. I think it might be beneficial for these who want to see how I did it. They might spot some shortcomings down the road or just get a better picture of what is it really about. Plus I would appreciate some feedback before I spend hours working on something that has no future, just because I was unaware of something 😄

[nicopap]

I did take a look at your repo. I've some feedback. Would you like to receive it in the form of issues/prs on IDEDARY/bevy_lunex_examples?

[IDEDARY]

Sure, of course. I would love to. You can either do issues on that examples repository or we can talk on discord, tag: idedary. From the time that I posted that I have been working on it on the actual official branch here, which is where the library will be located.