How should we drive runs of the Render Graph?

[cart]

I've recently been pushing for "RenderTarget entities as Render Graph drivers" over "Cameras as RenderGraph drivers", as I think RenderTarget entities map best to the primitives and patterns we've built. But I've given it more thought and I'm not so sure anymore.

I'm curious to hear what others think about these options.

Note that I don't want to discuss "camera/render target relative ordering" here because that feels tangential to the core problem. Assume that we will adopt a good way to order "drivers" relative to each other (ex: a simple way would be to give each driver an integer weight, sort the drivers by weight, and then run in that order).

RenderTarget Entities as Render Graph Drivers

• Godot uses this approach
• The general idea:
  • RenderTargets are entities with a RenderTarget component
  • Each RenderTarget corresponds to a "Sub RenderGraph" run each frame.
  • Each RenderTarget maps to a Window or Texture
  • Cameras are "just" render perspective configuration. nothing else.
  • Cameras can be "active" or "inactive". For a given render target, there can only be one active camera of a given type.
  • Each Camera entity maps to a single RenderTarget entity. Cameras are "typed" (ui, 2d, 3d) and RenderTargets will expect a certain set of camera types (based on the render graph being run).
  • There is a "default render target" that maps to the primary window.
  • Configuration like "render viewport", ClearColor, and post processing lives on the RenderTarget entity
• Handles "many camera types and perspectives in one graph run" nicely + consistently (ui, 2d, 3d). Can be extended indefinitely
• "Post processing" can be in any scope. 3d-only, 2d-only, final-product, etc. RenderTargets encapsulate a "global" view of a render and each camera type slots somewhere into that global view.
• Makes it easier to use hierarchy for "render target entity visibility". Spawning a scene "under" a RenderTarget could make it visible exclusively to that target, while still allowing you to move cameras that correlate to that target freely.
• Maps better to whats happening "logically", but at the cost of higher user-facing complexity

Cameras as Render Graph Drivers

• Unity uses this approach
• The general idea:
  • Cameras are "untyped" ... one camera == one graph run. UI / 2d / 3d / etc are all driven by a single camera "type"
  • if a camera is "enabled" it runs (and cameras are enabled by default)
  • cameras are configured via components. want a UI pass on this camera? add the relevant UI component
  • Each enabled Camera corresponds to a "Sub RenderGraph" run each frame.
  • Cameras specify the Window or Image they will render to.
  • Configuration like "render viewport", ClearColor, and post processing lives on the Camera entity
• How do we reconcile multiple cameras of different types (2d, 3d, ui, etc) per graph run?
  • We don't: Cameras with UI enabled generate their own internal UI projection. Whether a camera is 2d vs 3d is just a matter of configuration.
  • One way this is "nice" is that users won't need to manually spawn UI cameras ... a common gripe. But it also makes UI perspectives "different" and potentially less configurable
• How do we do "2d on top of 3d"?
  • multiple cameras rendering to the same render target. just make sure you handle per-camera entity visibility properly and disable UI rendering on the "first" camera!
• Blurs the line between 2d and 3d as they aren't explicitly sectioned off
  • cameras "just" render everything by default?
  • Could we still use "additive marker components" to select what each camera sees and draws? Will that complicate the user experience?
• More intuitive for users: rendering a new thing == new camera. Configuring a camera configures how it "views" the world (just like the real world).
• Easier to work with active cameras (all enabled cameras are "active" and map to a "target"). No need to query for each render target's active camera
• Managing multi-camera scenes is more complicated. How do you switch between active cameras in a scene?
  • You dont. Switching between perspectives would require either transforming the camera or turning cameras on and off
    • Could use the "virtual camera" pattern like Unity Cinemachine
  • Each camera owns its own "render config" (such as post processing), which means if you want to switch between multiple cameras and use the same config, you need to duplicate it (or use virtual cameras like Cinemachine).
  • Loading scenes with multiple cameras could be performance (or render correctness) foot guns. Double-spawning a default camera silently halves render perf.
• Makes it harder to use hierarchy for "render target entity visibility". Spawning a scene "under" a Camera would mean that by default it inherits the Camera's transform, so you can't freely move the camera. Users would need to disable transform propagation for the camera's children (edit: this could be easily resolved by adding a visible_hierarchy: Option<Entity> field to the camera).

Conclusion

This isn't clear cut. I think RenderTarget entities are a more flexible abstraction that better maps to whats happening under the hood (and the typed camera patterns we use today). But letting Cameras drive rendering makes some of the common cases easier + more intuitive. In short: im torn. I'm curious to hear what everyone else thinks.

[james7132]

I'm personally more used to the camera driven approach, but that might just be my prior Unity experience biasing it.
The primary two groups of users interacting with this are programmers dealing with the API directly and game designers via the editor. Having to set up multiple cameras, including a dedicated UI camera, can be rather unintuitive. We can obviously make a different editor interface than the underlying representation, but that causes a disconnect for the programmers interacting with it. Plus just having a "drop a camera into a scene" as game designer UX cannot be underestimated , without leaking the underlying abstraction to the user.

I'm not too fond of the idea of using the hierarchy to limit visibility beyond global visibility inheritance. IMO it's not all that intuitive. Imagine adding a first person-camera to the root of a player character, and suddenly the character's body is now only visible to that player? It also encourages "foldering" hierarchies which has runtime costs in other ways (i.e. transform propagation).

Post-processing is the part that I definitely think we should be able to have finer control over how it applies. However, this can be exposed as a thin config component in the app world and a more complete extracted state.

[cart]

I'm not too fond of the idea of using the hierarchy to limit visibility beyond global visibility inheritance. IMO it's not all that intuitive. Imagine adding a first person-camera to the root of a player character, and suddenly the character's body is now only visible to that player? It also encourages "foldering" hierarchies which has runtime costs in other ways (i.e. transform propagation).

The main reason to do this is that it enables you to spawn an arbitrary number of "scoped" scenes, each rendered to a different target, and have it "just work" without any additional configuration. Right now we only have per-entity RenderLayer visibility (with "cheap" support for 32 layers using bitsets). To handle rendering multiple "scoped" scenes to different targets, each entity in each scoped scene would need a manually configured RenderLayer. First, doing this from an editor would be nasty (especially if you want to render multiple instances of the "same" scene, each to a different target). And what if you need to render more than 32 "separate" scenes from separate perspectives? Being able to say "I want this specific scene instance to only be visible to this render target" is a powerful concept. And using hierarchy to represent scenes is currently my plan. Its how Godot's scenes and Unity's nested prefabs work.

[james7132]

Extracting the visible entities in this system would require traversing the hierarchy, which could incur heavy random reads/writes during extraction. Unless we have dedicated hierarchical storage, we could see some major performance degradation here. Is the idea to have some SceneId component embedded in every descendant of the root(s) here?

I can easily envision more than 32 scenes being used, or even heavily duplicated from a initial base scene. However, use cases with more than 32 different scoped views that isn't already covered via bitset exclusion is something I'm having a hard time visualizing.

[cart]

Unless we have dedicated hierarchical storage, we could see some major performance degradation here

This is definitely worth benchmarking. I'm betting that the cost will be "acceptable", but we can't really know without looking at the numbers. Obviously faster hierarchy is better. But I don't want to exclude hierarchy traversals from our design space, just because we don't have "optimal" performance yet. I do know that Godot uses this approach and im pretty sure they use normal pointers for hierarchy without any accelerated structures. We use indices (which can be slower than pointers, depending on the situation), but we also do at least have a chance of cache hits because we store Children components "next to each other'.

Is the idea to have some SceneId component embedded in every descendant of the root(s) here?

Tracking scene instances is a bit of an open question atm. Right now we store InstanceId-to-Entity mappings in a resource (but idk if thats what we'll do ultimately). I don't think it makes sense to mark every scene entity with a component. Rather, we would probably just mark a "scene root" and then define everything under it as "that scene".

I can easily envision more than 32 scenes being used, or even heavily duplicated from a initial base scene. However, use cases with more than 32 different scoped views that isn't already covered via bitset exclusion is something I'm having a hard time visualizing.

I think bitsets for "user-facing layers", such as marking "scenery" vs "player characters" is totally reasonable. Blender and Godot both use this pattern and I've found it useful. One case for "over 32 scoped scene renders" would be rendering Bevy UI components to textures, which are then used in a 3d world for per-chararcter ui (ex: displaying health bars or names above characters). I used that approach to render the player portraits in my game High Hat (which was built in Godot).

[cart]

Heres an illustration of what these approaches might look like for common use cases:

// RenderTarget-driven render-to-texture without UI
let render_target = commands
    .spawn_bundle(RenderTargetBundle {
        target: Target::Image(my_image_handle),
        ui_config: UiConfig::disabled(),
        ..default()
    })
    .id();
commands.spawn_bundle(Camera3dBundle {
    render_target: Some(render_target),
    transform: Transform::from_xyz(1.0, 2.0, 3.0),
    ..default()
});

// Camera-driven render-to-texture without UI
commands.spawn_bundle(CameraBundle {
    target: Target::Image(my_image_handle),
    transform: Transform::from_xyz(1.0, 2.0, 3.0),
    ui_config: UiConfig::disabled(),
    ..default()
});

Note that these would default to the "default render graph" because one wasn't manually specified in the bundles.

[CptPotato]

FWIW, I personally like the "RenderTarget as driver" approach, although I'm fine with both, I guess.

But I do agree with @james7132 that using the existing hierarchy for render target visibility can end up being confusing and/or slow.

Also: how would this approach look like if the render graph produces something other than an image?
(i.e. a render graph might run a compute pass to produce a mesh/buffer that is used in a different graph)

[cart]

Also: how would this approach look like if the render graph produces something other than an image?

I think this works reasonably well already with the current approach to custom render graphs / render graph drivers. The default graph is built to produce images, so if you want something different, you would define a new custom graph and add a driver to the "top level" / main graph. Likewise, you are free to extend existing graphs if you want to add logic that runs for each target.

[CooCooCaCha]

I feel like the RenderTarget driver approach is cleaner but either way is probably fine. What I'm not clear on is why, in the RenderTarget driver approach, does the render target need to be a separate entity? Couldn't you have a camera component and a render target component on the same entity? If that's the case then it seems similar in usability to the camera approach.

[cart]

What I'm not clear on is why, in the RenderTarget driver approach, does the render target need to be a separate entity? Couldn't you have a camera component and a render target component on the same entity? If that's the case then it seems similar in usability to the camera approach.

In the RenderTarget driver approach, there is a many-cameras-to-one-render-target relationship. Adding a render target component to a camera entity breaks that model, unless you build some sort of "render target component merging across cameras" approach, which feels confusing to me. I think with a "point your cameras to your render target entity" approach, you could pick one camera to be the "render target entity", but I feel like that conceptually muddies the waters / isn't a pattern we should recommend generally.