RenderLayers performance drop

[HaHa421]

When increasing the number of RenderLayers and spawning one Camera3d per layer,
performance drops significantly, even in minimal scenes.

In the following test(based on examples/3d/3d_scene), where only one camera sees the spwaned cube (RenderLayers 0) , I measure:

Layers  |    Cameras    |    FPS
1       |       1       |   ~900
2       |       2       |   ~720
5       |       5       |   ~425
10      |      10       |   ~270

test:

//! A simple 3D scene with light shining over a cube sitting on a plane.

use bevy::{
  prelude::*,
  render::view::RenderLayers,
  window::PresentMode,
};

#[derive(Resource)]
pub struct FpsTimer(pub Timer);

const ADDITIONAL_LAYERS:usize = 1;

fn main() {
    App::new()
        .add_plugins(DefaultPlugins
          .set(WindowPlugin {
            primary_window: Some(Window {
                #[cfg(not(target_os = "android"))]
                present_mode: PresentMode::Immediate,
                #[cfg(target_os = "android")]
                present_mode: PresentMode::Fifo,
                ..Default::default()
            }),
            ..Default::default()
          })
        )
        .add_systems(Startup, setup)
        .add_systems(Update, 
          fps_counter,
        )
    .run();
}

/// set up a simple 3D scene
fn setup(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<StandardMaterial>>,
) {
    
    // cube
    commands.spawn((
        Mesh3d(meshes.add(Cuboid::new(1.0, 1.0, 1.0))),
        MeshMaterial3d(materials.add(Color::srgb_u8(124, 144, 255))),
        Transform::from_xyz(0.0, 0.5, 0.0),
        RenderLayers::layer(0),
    ));
    // light
    commands.spawn((
        PointLight {
            //shadows_enabled: true,
            ..default()
        },
        Transform::from_xyz(4.0, 8.0, 4.0),
    ));
    // camera
    commands.spawn((
        Camera {
          order: 0,
          ..default()
        },
        Camera3d::default(),
        Transform::from_xyz(-2.5, 4.5, 9.0).looking_at(Vec3::ZERO, Vec3::Y),
        RenderLayers::layer(0),
    ));
    
    for i in 0..ADDITIONAL_LAYERS {
        commands.spawn((
            Camera {
              order: (i+1) as _,
              ..default()
            },
            Camera3d::default(),
            Transform::from_xyz(-2.5, 4.5, 9.0).looking_at(Vec3::ZERO, Vec3::Y),
            RenderLayers::layer((i+1) as _),
        ));
    }
    
    commands.insert_resource(FpsTimer(Timer::from_seconds(1.0, TimerMode::Repeating)));
}

pub fn fps_counter(time: Res<Time>, mut timer: ResMut<FpsTimer>) {
    timer.0.tick(time.delta());
    if timer.0.finished() {
       println!("FPS: {}", 1.0 / time.delta_secs());
    }
}

[HaHa421]

Btw , I tested a naive Core3d/Core2d subgraph parallel execution using thread::scope, reordering CommandBuffers at scope exit: it didn't provided the expected results (performance is even worst, and the commands don't seem to be sent in the correct order to
the GPU, I might provide the code when needed).

[HaHa421]

Anyway, here's what I tried:

1. Core3d and Core2d declarations need to be moved from
   bevy_core_pipeline to bevy_render
2. I modified RenderContext fields visibility to be able to create a RenderContext
   from an existing RenderContext (render_device, is_serial,and diagnostics_recorder) and added new2, please
   forgive lack of creativity in function naming:

in bevy_render/src/render/mod.rs(365):

pub struct RenderContext<'w> {
    pub render_device: RenderDevice,
    pub command_encoder: Option<CommandEncoder>,
    pub command_buffer_queue: Vec<QueuedCommandBuffer<'w>>,
    #[cfg(not(all(target_arch = "wasm32", target_feature = "atomics")))]
    pub force_serial: bool,
    pub diagnostics_recorder: Option<Arc<DiagnosticsRecorder>>,
}

impl<'w> RenderContext<'w> {
    /// Creates a new [`RenderContext`] from a [`RenderDevice`].
    pub fn new2(
        render_device: RenderDevice,
        #[cfg(not(all(target_arch = "wasm32", target_feature = "atomics")))]
        force_serial: bool,
        diagnostics_recorder: Option<Arc<DiagnosticsRecorder>>,
    ) -> Self {
        Self {
            render_device,
            command_encoder: None,
            command_buffer_queue: Vec::new(),
            #[cfg(not(all(target_arch = "wasm32", target_feature = "atomics")))]
            force_serial,
            diagnostics_recorder: diagnostics_recorder,
        }
    }
}

3. in RenderGraph::run_graph
   bevy_render/src/renderer/graph_runner.rs(234)

let run_sub_graphs = context.finish();
let run_sub_graphs_len = run_sub_graphs.len();
if run_sub_graphs_len > 0 {
use crate::Core3d; // these have been moved to bevy_render
use crate::Core2d;
use crate::render_graph::RenderSubGraph;
fn is_core3d_or_core2d(s: &InternedRenderSubGraph)->bool {
//didn't investigate further, might be suboptimal  
  s.as_dyn_eq().dyn_eq(Core3d.intern().as_dyn_eq())
  || s.as_dyn_eq().dyn_eq(Core2d.intern().as_dyn_eq())
  || s.as_dyn_eq().type_id() == Core3d.intern().as_dyn_eq().type_id()
  || s.as_dyn_eq().type_id() == Core2d.intern().as_dyn_eq().type_id()
}
//Doesn't work as expected, slower than serial/sequential anyway
if true && is_core3d_or_core2d(&run_sub_graphs[0].sub_graph) {
  //println!("running Core3d-Core2d graphs");
  use std::thread;
  let results = Arc::new(Mutex::new(Vec::with_capacity(run_sub_graphs.len())));
  
  //use crossbeam::thread;
  use std::sync::Arc;
  use std::sync::Mutex;
  thread::scope(|s| {
  
      for (idx,run_sub_graph) in run_sub_graphs.into_iter().enumerate() {
        
        let mut render_context_ = RenderContext::new2(
          render_context.render_device.clone(),
          
          #[cfg(not(all(target_arch = "wasm32", target_feature = "atomics")))]
          render_context.force_serial,
          render_context.diagnostics_recorder.clone(),
        );
        let sub_graph = graph
            .get_sub_graph(run_sub_graph.sub_graph)
            .expect("sub graph exists because it was validated when queued.");
        let results = results.clone();
        s.spawn(move || {
          match Self::run_graph(
            sub_graph,
            Some(run_sub_graph.sub_graph),
            &mut render_context_,
            world,
            &run_sub_graph.inputs,
            run_sub_graph.view_entity,
          ) {
            Ok(_)=>results.lock().unwrap().push((idx,Ok(render_context_))),
            Err(err)=>results.lock().unwrap().push((idx,Err(err))),
          }
        });
      }
   });
   let mut r = results.lock().unwrap();
   let mut results = vec![];
   std::mem::swap(&mut *r,&mut results);
   assert_eq!(run_sub_graphs_len,results.len());
   results.sort_by(|(idx,_),(idx2,_)|idx.cmp(idx2));
   for (idx,(_idx,render_context_)) in results.into_iter().enumerate() {
     assert_eq!(idx,_idx);
     let (command_buffers,_,_) = render_context_?.finish();
     for command_buffer in command_buffers {
      render_context.add_command_buffer(command_buffer);
     }
     
   }
} else {
  for run_sub_graph in run_sub_graphs {
      let sub_graph = graph
          .get_sub_graph(run_sub_graph.sub_graph)
          .expect("sub graph exists because it was validated when queued.");
      Self::run_graph(
          sub_graph,
          Some(run_sub_graph.sub_graph),
          render_context,
          world,
          &run_sub_graph.inputs,
          run_sub_graph.view_entity,
      )?;
  }
}

To summarize, performance is worst and display is inconsistent (inconsistent from one frame
to the other: camera order doesn't seem to be preserve, lol "against all odds", meanwhile works when only one camera sees objects ala the exemple above)

[janhohenheim]

Oof, that's bad. Considering that the average industry-standard first-person game will have 3 cameras (one UI, one view model, one world model), this is a huuuge performance drop :/

[JMS55]

Please profile (https://github.com/bevyengine/bevy/blob/main/docs/profiling.md) your app and see where the bottleneck is.

[HaHa421]

I consider the example above to be the test app: it is reduced to what one might consider to be minimal so that the bottleneck can be revealed. I'm not sure where profiling would occur in this setup, but it looks like profiling Bevy's internals might be more insightful. Any suggestion is welcome.

[janhohenheim]

I think Jasmine is asking you to check with tracy which systems are causing the slowdown :)

[HaHa421]

Well, I don’t plan to profile the modification I tried, as I have the impression that it wouldn't work without isolating each camera in its own RenderApp — and thus a dedicated RenderWorld. My insights into Bevy internals right now are more of a high-level view than deep technical knowledge.

[HaHa421]

With the parallel implementation I “showcased”, one open question I have concerns the possibility that the same system might be executed concurrently, even though it requests exclusive access to ResMut resources. Rust’s borrow checker should enforce exclusivity at compile time, but Bevy’s internal orchestration might rely on unsafe assumptions that aren’t explicitly surfaced. I haven’t explored this exhaustively, but this could help explain some of the visual inconsistencies I’ve observed — despite the absence of unsafe code in my own implementation.
(Side note: English is not my native language, and I discussed this wording with Copilot over breakfast — it seems to capture the sense I’m trying to express here.)