Parallelism of apply_system_buffers (stageless scheduling)

[smessmer]

So it seems in bevy 0.10, the new way to handle systems that need to wait for another system's commands to be applied before running is to add a separate apply_system_buffers system and make sure they're ordered correctly.

What are the performance implications of this? Afaik apply_system_buffers is an exclusive system, so we could end up with lots of different plugins in the same game, all scheduling their own apply_system_buffers, causing the game to have a frame with lots of exclusive systems scheduled and little parallelism.

Actually, for this use case, each plugin could be separate and each plugin only cares about its own commands being applied, it doesn't care about applying commands from other plugins. Can we improve parallelism here?

A couple of options jumping to my mind:

Option 1: Automatically de-duplicate calls to apply_system_buffers.

For example, users wouldn't manually add a apply_system_buffers system, but instead define the dependencies with something like

// Plugin 1
app.add_system(my_system_1.after_buffers_applied(other_system_1))

// Plugin 2
app.add_system(my_system_2.after_buffers_applied(other_system_2))

which would run other_system first, then apply its buffers, and then run my_system.
If there are multiple such dependencies, the scheduler can try to be smart and schedule them so that other_system_1 and other_system_2 run first, in parallel, then there is just one de-duplicated call to apply_system_buffers, and then my_system_1 and my_system_2 run in parallel.

Option 2: Separate command buffers

Each plugin could have their own commands buffer and then add a system that just applies their own commands in a system that isn't world exclusive but only locks the command buffer for that specific plugin. This would allow other plugins to run while a plugin is applying its commands.

[SkiFire13]

and then add a system that just applies their own commands in a system that isn't world exclusive but only locks the command buffer for that specific plugin.

How can you apply commands without exclusive world access though?

[smessmer]

Yeah good point, I hadn't thought about that. I guess to make this happen, it would require us to introduce a concept of a plugin-specific sub-world containing entities exclusive to the given plugin and its own command buffer. It could work for some plugins but likely not for others and might not be worth the complexity.

The reason I was thinking in that direction is actually not for commands accessing entities, but for commands inserting resources. I have a couple of startup systems that load assets and insert an asset catalog (e.g. a struct with lots of Handles) as a resource, and other startup systems that create entities using those assets. I have several plugins using this pattern and each needs to make sure its own resources are inserted before the startup system creating entities runs but they don't care about the resources of other plugins. Those resources are private structs in the plugin module and cannot be accessed from any code outside of the plugin. So it sounded like we should be able to insert those in parallel. For this specific case, a sub-world would solve the problem.

[SkiFire13]

I have several plugins using this pattern and each needs to make sure its own resources are inserted before the startup system creating entities runs but they don't care about the resources of other plugins.

Is this really worth the effort for startup systems? Efficiency shouldn't matter that much for them since they're one-off things.

Those resources are private structs in the plugin module and cannot be accessed from any code outside of the plugin. So it sounded like we should be able to insert those in parallel.

The metadata is still shared though. There might be a lock-free way of doing this, but I doubt it's worth it just for resources.

[alice-i-cecile]

For resources at least, we should be able to do non-blocking creation. See the discussion and ideas in bevyengine/rfcs#30

[alice-i-cecile]

Option 1 is discussed in #7838, and is something the ECS crew is interested in implementing.