Results of Multi-world research and next steps

[ElliottjPierce]

History of Multiworld?

A World represents an entire ECS ecosystem. But what if you want multiple ecosystems? This has been a long-requested feature in Bevy. It is possible today to use multiple worlds together. That is in essence what SubApps are. But just about everybody agrees that SubApps and Apps are pretty clunky. They have inflexible points of communication, can't really be created once an app starts, require mapping entity ids between worlds, etc.

As a rough timeline:

• 2020 Bevy starts
• 2021 Sub-worlds discussed: rfc
• 2021 Disjoint worlds requested: Multiple worlds for isolated simulations #2237
• 2022 SubApps fill the void (mostly): bevy 0.6
• 2023 Multiworld given a name: Multi-world support #7893
• (lots of assorted decentralized discussion)
• 2024 A brute force attempt: Per world access lists and utils #17198
• Feb 2025 Discussion for how to proceed: First Class Multi-World Bevy #17937
• Multiworld requested again for editor work and other assorted use cases
• June 2025 I begin prototyping Multiworld ECS

A lot has happened, and while just about everybody thinks multiworld should be a reality, no consensus has been reached. This is largely because of Bevy's (good) ideal of keeping PRs relatively small. Multiworld is a HUGE feature. It would take many small PRs to add multiworld, and nobody wants to review (or merge) a PR like this without understanding the endgame. And nobody can understand the endgame without seeing a prototype in code. And round and round we go lol.

I set out to build my own ECS engine to prototype many of my wish-list bevy features that weren't coming any time soon (Multiworld, archetype invariants, systems as entities, trait query, schedule-less, etc). Ultimately, I don't think I want to publish the engine on its own when it would be so much nicer to just get it into Bevy, but doing that is hard. I may change my mind if my vision becomes fundamentally different than Bevy's, and that's ok. But for now, I want to share what I've learned about multiworld in the hopes of Bevy being able to add it soon(er)/eventually.

Why multiworld

This has been discussed many times, so I'll try to keep this brief. Advantages of multiworld as opposed to a single world include:

1. Parallel structural changes (spawning, adding/removing components, etc.) If there were only one world, spawning something (or any command) would not be applied until the entire frame finished rendering! That would be unacceptable, hence, SubApps.
2. Sandboxing. Having multiple worlds does nothing to prevent malicous acting, but it makes it harder to cause accidental logic issues. If game logic, rendering, physics, player i/o (input and visuals), etc are all in different worlds, it becomes much harder for errant render code, for example, to affect game logic.
3. Parallel extraction. Right now, each SubApp's extraction happens in sequence. That's fine internally because bevy itself only has one SubApp, the render world. But for real applications that may have a physics world (at least) as well as others, this is room for improvement.
4. Concurrent schedules. Ideally multiple systems run in parallel. In the main app, there are First, PreUpdate, Update, etc. But the schedules for the render world are different. Both need to run in parallel. If we were only using one world, PreUpdate for game logic could only run once First for render logic was finished. Lots of bottlenecking potential.
5. Adding and removing worlds at runtime. For example, loading a new world would be very helpful for the editor. Each scene could be loaded as an independent world for inspection, editing, etc. (This is probably not a good idea in practice, but it's a good example and has been discussed before.)

There are also some things that multiworld would help with but wouldn't fix. These are related non-advantages:

1. Multiple resources. There are very few times when an app should logically only have one copy of a component. In those cases, they considered a Resource. Aside from Time and other data, there's very few things that are logically per-world. For example, resources are sometimes used to represent data about the player. That's fine for single player, but breaks down for multiplayer. End users can get around that, but it's much harder for plugins. I would guess that's why some plugins, like input plugins, are available through resource and component APIs. Multiworld isn't a good solution for this though. It would help, but it's impractical to have a separate world per UI menue, per player, etc. A better solution is a more generalized Res, but that's a topic for a different discussion.
2. Multiple states. States are resources, and the same things apply here too. But nobody, especially plugins, can use states for much. States can't be used for UI; what if there are multiple UIs? States cant be used for players; what if there are multiple players? States can't be used for game states; what if there are multiple states, like for multiplayer or a replay view? You get the idea.
3. Generalized sandboxing. Ultimately, separate processes are needed for complete separation. We don't need a Kubernetes multiworld solution.

Defining Multiworld

There have been debates over what multiworld really means. It doesn't have to be implemented by literally getting multiple World's to communicate. Ultimately, a multiworld solution is an ECS where entities can have multiple components in disjoint storage systems, such that structural changes to one storage can be parallelized with all other storages. I know that's a bit of a mouthful. Put another way, any solution that fully satisfies the 5 advantages of multiworld listed above is a multiworld solution.

Multiworld is... awkward

Multiworld solutions have a few fundamental issues. First, it no longer makes sense to think of systems as "per-world". A system can access data from multiple worlds. This is not a problem; it is the solution to advantages 3 and 4. But it means that systems don't really have a world. They can have a base world, where they name other worlds based on those worlds' relation to the base world. But fundamental structures like schedules, executors, etc can no longer work under the very convenient premise that there is only one world.

Access checking is also an issue. It already takes so much work to coordinate systems in parallel that the benefits of parallelism are almost lost. The more we try to parallelize, the more that parallelism costs in overhead. Bevy has recently moved in the direction of "parallelize less, faster" with #19143. Adding something like MultiworldComponentId would "parallelize more, slower". This is easily the biggest blocker on multiworld today.

Solutions

This is hard. Really hard. But, here's what I've come up with. I just defined "multiworld solution" as satisfying the 5 advantages above, so lets go through them one at a time.

Let's start with advantage 4: parallel schedules. This is the easiest one. The solution: get rid of schedules 😱. Let's go schedule-less. The simplest way to do this is to 1) Buff system sets to allow more ordering configuration there. 2) Allow ordering in observers, etc. 3) Allow a system to queue another system onto the executor as it is running. 4) Let those queued systems also be ordered. 5) Have one, singular, per-app, executor. 6) Make schedules just be the act of queueing to the executor multiple ordered systems. Tada! That's the easy one. Bonus: Combine this with async systems, and the whole update cycle could be a loop with some "await sleep" and "queue Update systems" logic. (Oversimplification).

Next, lets look at advantage 1: parallelized structural changes. That means having multiple copies of the Storages, Archetypes, Entities, etc fields of World. A structural change, (ex: spawn an entity) only needs to access one of these storage groups/sub worlds/worlds/data domains (the name isn't important right now, and will probably depend on how this is implemented). I'll call it a world for now. If there are multiple worlds, structural changes to one world can be parallelized with structural changes to other worlds. The point is, we need multiple copies of the "data storage" part of the ECS.

Advantage 2: Splitting logic between worlds (ex: render world, logic/main world, physics world, etc). The implementation challenge here is that the worlds described in the previous paragraph need to communicate. Not only that, but if we want the worlds to be able to store any data, worlds need to have components, not the other way around. (In other words, the same component type may be registered in multiple worlds at the same time under different ids.) That means 2 things. First, the "component" data needs to also be part of these worlds/sub-worlds/storage groups/whatever-its-called. That's easy enough. Second, there must be a way to map entities in one world to entities in another. That's easy with a hashmap, but we can do better. More on that later.

Advantage 5: Adding/removing worlds at runtime. This is also easy on paper. Just make worlds entities. Each world can have child worlds that serve some function for their parent world. Ex: The main world could have a render and physics world. But this has a big problem. As an example, imagine an asset world that stores all the assets for the main world. Well, the main world needs to access that world, but so does the render world. Ideally, the render world shouldn't need to impose a read on the main world (which would block structural changes) just to read from the asset world. This means we need a centralized world storage type. Maybe App? It would probably end up having something like RwLock<HashMap<WorldId, Arc<UnsafeCell<World>>>>. Then, we store the world id and/or a Weak<UnsafeCell<World>> as a component on a world. The tricky part here is managing removing the worlds. When the main world ends, so should its render world. I am yet to find an uncompromising solution to this. We would also need to at least warn when tampering with these WorldLink components. Designing this is hard, but this is the general idea.

Advantage 3: Parallel extraction phase. This is not easy on its own, but with the other 4 issues resolved, we can just make extraction a schedule. Tada! One of the details here is that we would need query joins between world. Ex: "query for component A from world 1 and component A from world 2; then, copy the data from 1 tot 2." This is not too hard to do, but it is worth mentioning. We need to make inter-world communication low friction.

Bonus 1: Remember how I said we could do better than a hashmap for mapping entities between worlds? Consider the case where a world exists purely to mirror another world in order to defer some work (ex the render world just has entities that "mirror" what to render for their corresponding main world entity). Of course, the serving world will also have its own entities. One thing we can do here is entity paging (see #19430). Basically, this lets us efficiently start the entity allocator for a world at any EntityRow we like. For example, the main world could allocate entities between dis 0 and 1 million, and the render world could allocate entities between ids 1 million and 2 million. Of course, the numbers could change (we have 4 billion to work with), but the general idea is that, when a world directly mirrors the entities of exactly one other world (very common), start its entity allocator where its source world's allocator (the one it's mirroring) ends. This also means worlds would need to declare their max entity count, but we can have a reasonable default. Then for example, the render world could spawn an entity at the same id as any entity in the main world. Very convinient! There are other advantages of entity paging, but this is the big advantage for multiworld. Practically speaking, this means main and render world entities could share the same id space!

Bonus 2: Looking at my solution to advantage 4 again, it means there would be one, per-app system executor. That means we need to include the WorldId in the access checks somehow. We could do a WorldComponentId and parallelize over that. We could also just make communicating between worlds really easy and just parallelize per world instead of per component, which may be even faster. Ultimately, the answer here is benchmarking, and lots of it. We could even make this configurable. There's lots of options here, and we really can't afford a regression in access checking.

My solution

For my prototype ECS, what I settled on is communicating between multiple Worlds, where each World is Arced on an App, and each App has its own executor. My solution is very much evolving and is not in a sharable state (scarce documentation, rapidly changing details). However, I've done enough work to know that this design should work and will satisfy most, if not every, use case for multiworld.

Roadmap

It's going to take a lot of work to get here. A lot of work. And I'm not even sure if "here" is where Bevy wants to go. Lots of discussion needed! That said, the next real steps to making multiworld a reality is schedule-less systems and a per-app executor as explained above. Most of the tricky parts are centered in that implementation, and others have done much more research there than I have. We can also look into entity paging at the same time too.

I'm sharing this because the broad phase design work for multiworld in my ECS is pretty much complete. I'm hoping that sharing these ideas will let Bevy start heading in the direction of multiworld or help Bevy articulate a case against pursuing multiworld further if this is not the direction we want to go. At the end of the day, all this is still very much just an idea, and more discussion will reveal how to proceed from here.

[EmileGr]

One of the details here is that we would need query joins between world. Ex: "query for component A from world 1 and component A from world 2; then, copy the data from 1 tot 2." This is not too hard to do, but it is worth mentioning. We need to make inter-world communication low friction.

This helped me a lot to understand how communication between worlds would work. Particularly if entities share the same id across worlds then it becomes fairly simple as you've pointed out. Obviously the copying would have to happen safely and soundly. I guess as long as there's only one executor per app then Bevy's existing safety guarantees should ensure worlds only copy data when its safe to do so, right?

That type of "sync point" between worlds will be important. Particularly for rendering, you don't want the main world to get too far ahead of the render world in situations where your main world completes a frame much faster than your render world. The main world should start blocking if it gets more than 2 or 3 "frames"/update cycles/whatever you want to call it ahead of the render world.

[ElliottjPierce]

as long as there's only one executor per app then Bevy's existing safety guarantees should ensure worlds only copy data when its safe to do so, right?

Exactly. This is one of the reasons there must only be one executor for the entire app.

That type of "sync point" between worlds will be important. Particularly for rendering, you don't want the main world to get too far ahead of the render world in situations where your main world completes a frame much faster than your render world. The main world should start blocking if it gets more than 2 or 3 "frames"/update cycles/whatever you want to call it ahead of the render world.

Right. By making sync points systems, and making system ordering more powerful, it should be pretty easy to add these "wait for the render world" points. We don't want to lock in this behavior by default though. For example, an asset world shouldn't need to block the main world's execution, even if it's taking a little longer to load an asset than planned.

[EmileGr]

We don't want to lock in this behavior by default though. For example, an asset world shouldn't need to block the main world's execution, even if it's taking a little longer to load an asset than planned.

As long as I'll have the ability to create sync points as I please then I'll be fairly happy with this proposal for multi-worlds. I do agree with your asset loading example to a degree. I'd really love that behavior for an editor. Should let you get up and running quite quickly while the heavier parts of the scene load in the background.

For the runtime of a game, however, I could easily see that causing problems. Like, suppose your character controller loads in before your floor. With the main world and/or a physics world running at full speed you could clip through the floor (possibly even at random if the order that assets are loaded in isn't guaranteed). But I suppose as long as developers are aware of how it works and/or have the option of blocking until all assets have been loaded then it won't be a problem.

[alice-i-cecile]

Overall I like this, although I'll quibble with details and think that this post would have been stronger with clearer concrete use cases to motivate it :)

Just make worlds entities

Why use this design, rather than a Vec<World> inside of an App?

Parallel schedules

I think your implementation plan is too complex: observer ordering etc is not needed. Simply swapping to a system set-first approach and removing the end-of-schedule flushes would fix things.

Access checking could also be accelerated by doing a two stage check: first for world, then for data within that world. This pairs especially nicely with "maintain multiple copies of the storage infrastructure".

With an entity-partitioning flavored solution, you could also allow systems to access data from multiple "worlds" / "partitions" at once, making extraction and other forms of communication much easier.

single executor

Fully agree here. You need to have central orchestration. This is a big part of why I don't like our existing "one executor per schedule" model, and why I don't think that SubApp (with their own runners etc) is the right model at all.

[ElliottjPierce]

Overall I like this, although I'll quibble with details and think that this post would have been stronger with clearer concrete use cases to motivate it :)

Yeah, this is still early, very broad strokes. But I'm glad the general idea doesn't sound too far fetched.

Just make worlds entities

Why use this design, rather than a Vec<World> inside of an App?

We will need to store the actual Worlds in some structure on the App, yes. The worlds as entities part would probably look more like a struct WorldLink(WorldId) component. The "as entities" part is flexible (obviously). But we need some way to, for example, tell a render world which world it is rendering.

Parallel schedules

I think your implementation plan is too complex: observer ordering etc is not needed. Simply swapping to a system set-first approach and removing the end-of-schedule flushes would fix things.

Agreed. My point is just that under a "single executor" paradigm, running observes, for example, is effectively a "schedule". There's lots of potential here, but you're absolutely right. One step at a time.

Access checking could also be accelerated by doing a two stage check: first for world, then for data within that world. This pairs especially nicely with "maintain multiple copies of the storage infrastructure".

Yes! This is what I'm doing in my prototype. There's going to be lots of access list changes in Bevy's future. At the least, components as entities will force us to get rid of the current FixedBitSet scheme. Changing to world -> component locking is, I think, the best direction, but changing this could definitely cause butterfly effects.

With an entity-partitioning flavored solution, you could also allow systems to access data from multiple "worlds" / "partitions" at once, making extraction and other forms of communication much easier.

That's the hope!

single executor

Fully agree here. You need to have central orchestration. This is a big part of why I don't like our existing "one executor per schedule" model, and why I don't think that SubApp (with their own runners etc) is the right model at all.

I think working towards generalized schedules and single executors are the next steps here. Bonus points IMO if we just "make Forte be the executor". But I don't know enough about Bevy's plans there to know if that's viable. But it would be nice.

[NthTensor]

Brain dump about "parallelized structural changes". This is the point that concerns me most.

Most structural ops are clearly world-local; this includes adding and removing components, or any sort of archetype move. Spawns are different if all worlds share an ID space, but assuming you can claim an entity ID in parallel then moving it into an archetype is world-local and therefore also Parallelizable.

But what about despawns? When you despawn an entity, you have to do an archetype move in every world before you release the entity ID. Many designs also rely on entities continuing to exist during the runtime of a system, or even between systems.

I would expect despawns to work like this in a multiworld scenario:

• Just like spawns, despawns are world-local.
• Despawning an entity clears the archetypes on that entity in a specific world, and marks it for GC.
• At the end of a frame there is a a cross-world sync point where we clean up the data of marked entities.
• We can then release the entity ID for all the GCed entities and start the next frame.

This would allow us to set up the render world to be "one frame behind" the simulation world, but still share the same entity ID space. Despawning an entity in the sim world would keep it (and it's data) around until the render world was through with it.

I suspect there may be a few other operations that are not world-local, for which we will need to defer work to a once-per-frame cross-world sync. Does this seem reasonable?

[alice-i-cecile]

Does it make sense to use a ref-counting style for despawning, where you need to despawn an entity in all of the worlds it's been spawned in before it's fully cleaned up?

[NthTensor]

Interesting! That would let us despawn render-world stuff as part of extraction, which I like.

What does using an entity from world_1 in world_2 look like? Are we going back to spawn_reserved? Personally, I'd prefer if you didn't have to spawn it again. If you have an Entity IMO you should just be able to use it, no matter where it was spawned initially.

I ask because then, if we used ref-counting, you get a situation where you spawn an entity once and have to despawn it N different times. Feels odd. Also, would this mean you could despawn an entity (but not actually free the ID because of ref-counting) and then add components right back to it? We could just "pretend" that it was de-spawned, and refuse to allow you to use that entity in that world going forward.

By ref-counting, I assume you mean something like:

• Put an atomic ref counter on each entity.
• When you add the fist component to an entity in a world, you increment the counter.
• When you remove the last component from the entity in a world, you decrement the counter.

Then spawning would be:

1. Claim an entity ID.
2. If components were supplied, add them to the entity (and increment the ref counter).

And despawning would be:

1. If the entity has components in this world, remove them (and decrement the counter).
2. Check if the counter is zero, and if it is then release the ID.

I think there would be a subtle race condition if you de-spawned an entity in world_1 at the same time you were trying to add components it in world_2. Adding components to an entity spawned on a different world would have to be fallible. Unless we make that a specific call (like spawn_foreign or import_entity or something) I guess this would make all component-inserts fallible. But I guess they already are; if your generation is is out-dated inserts fail, and this isn't much different.

All of that seems workable. But the API will need consideration.

[ElliottjPierce]

This ref-counting solution works. It was one of the things I white boarded out trying to think this through. Here's my issues with it:

• Putting any kind of atomic or lock on each entity id, I feel, would cause pretty bad cache contention. We still need to read the generation for things like query gets. If we put the generation and this ref count in the same buffer, there's going to be lots of fighting between threads I would guess.
• The "claim an entity ID" (and releasing it) fully concurrently is really slow! That would require a fully concurrent entity allocator. I've built that before, for remote reservation, but that version only needed concurrent allocations, not concurrent frees. That was already much slower (about 2x) but adding concurrent frees makes ConcurrentQueue really the fastest implementation, which was more like 4x slower than today IIRC.
• Directly sharing entities between worlds would get confusing I think. One entity in multiple worlds is just kinda strange. I know that's not a substantive issue, but still.

My design is really much simpler. Most of the ideas going around have occurred to me too, but I think a simpler solution exists; though these would work too.

My design is not to share entities between worlds; just to make it possible to guarantee that two entities in different worlds happen to share the same EntityRow. That would allow super fast query joins without needing the mapping step.

Here's a case study. Lets say we have two worlds world_1 and world_2.

• We spawn an entity in world_1 and add a component Foo.
• Now we "extract" that entity to world_2. It's not already spawned in world_2, so we construct an entity at that row. Then we add a copy of the Foo component to it too.
• At this point, an entity has Foo in world_1 and another entity, that happens to have the same row, has Foo in world_2.
• We could continue adding and removing components from the entities in each world full in parallel.
• When the next extract comes, we don't need to re-construct the entity; we can just update its component values. But if the world_1 entity changed (ex: despawned and respawned) we would just despawn the mirror entity in world_2 and re-construct it as before.

---

Pros of single entity allocator (your idea)

• Lets entity id space be shared easier.
• Potentially can even skip an extraction step if the renderer just holds a lock on the "render type" components like Mesh3d and we defer changes to that storage fragment until the end of the frame.

Cons of single entity allocator

• Cache contentions
• Very slow (comparatively) spawns and despawns

Pros of multiple entity allocators with entities that happen to share a row (my idea)

• Very fast for one-world systems
• Crazy simple
• We can pretty much just reuse World and link worlds together

Cons of multiple entity allocators with entities that happen to share a row

• Eats into the id space a little. Ex: with a tree depth of 2 (main and render world) each world could get 2 billion entities instead of 4 billion.

I think that's a fair representation IIUC, but add more pros and cons if you think of them.

[NthTensor]

I see. So this is actually much more similar to what we had before, where we would manually spawn entities in one world with row/index matching entities in another world. The chief difference is then that we use paging to avoid cross-world ID conflicts, rather than despawning the entire render world and rebulding it from scratch.

It seems like we will need both rows and generations to match across worlds:

1. I spawn an entity 1:1 in world_1. I add components to it.
2. I do an extract, and spawn a corresponding entity 1:1 in world_2, making the rows match.
3. I despawn 1:1@world_1, time passes, I spawn more entities the row is eventually recycled as 1:2@world_1.
4. It's time to do another extract. I should not treat 1:1@world_2 as corresponding to 1:2@world_1, they are different entities even though they have the same row.

Do I have that correct?

[ElliottjPierce]

Pretty much! Though, I would argue that while it would make sense to make the generations match, it's not technically necessary, as long as each time the main world entity's generation changes, the render world completely refreshes its copy of the entity's data.

And it's just an idea. I just think it would be simpler and faster than a single entity allocator IMO.

[NthTensor]

Points of interest from discussion with @alice-i-cecile

Sharing the entity id space across worlds and allowing the same entity to have different components in different worlds is a very appealing idea.

This finally made the proposal click for me. The idea isn't "lazily reuse entity ids to save a mapping step" like we used, the idea is "the world is determined by the component, not the entity"

I think that framing this as "multiple worlds" is going to get confusing to users very quickly. You've convinced me that partitioning and mutliple worlds are basically equivalent

I agree that it might make sense to frame this as "shards" or "partitions" of a world, rather than changing the semantics of worlds.

[ElliottjPierce]

Points of interest from discussion with @alice-i-cecile

Sharing the entity id space across worlds and allowing the same entity to have different components in different worlds is a very appealing idea.

Agreed. It was originally conceived here, but we need entity paging to make it work.

This finally made the proposal click for me. The idea isn't "lazily reuse entity ids to save a mapping step" like we used, the idea is "the world is determined by the component, not the entity"

I'm so sorry, but I don't think we're on the same page here. This is really hard to both understand and explain. I'm not proposing that we have each component declare which "sub-world"/"storage fragment" it lives in. Like, we could have Mesh -> "main", RenderingMesh -> "render world", etc.

To be clear, that will work, and I spent a while considering it. It just breaks down when you ask for two render worlds, for example. And yeah, that would be weird, but we should make room for that flexibility IMO. The other issue with it is that allocating an entity id would need to be fully concurrent, which is possible (see #18670), but it's much slower, and you need to manage the cache contention when despawning and incrementing entity generations.

What I'm proposing is pretty much "lazily reuse entity ids to save a mapping step". I think that's the most flexible. See my response to your other comment for how I'm envisioning that working.

I think that framing this as "multiple worlds" is going to get confusing to users very quickly. You've convinced me that partitioning and mutliple worlds are basically equivalent

I agree that it might make sense to frame this as "shards" or "partitions" of a world, rather than changing the semantics of worlds.

100% agree here. There are lots of designs that fit the 5 advantages definition I gave. I think "shards" or "partitions" would be a good name for the "components pick their storage shard" solution. But if we instead want to map between entity ids in self contained Worlds, I prefer the multiple "world"s naming. But yeah, this will probably evolve overtime.

[NthTensor]

What I'm proposing is pretty much "lazily reuse entity ids to save a mapping step".

Yeah, I think that's what Alice was getting at. She was thinking "lets just add the world ID to the archetype table". Then if you have two worlds and one component A you get two tables, one for each world: A+world_1 and A+world_2 and you can do all the normal stuff you do with archetype tables. We use the same entity ID space for both.

When she says "the world is determined by the component, not the entity" she means the archetype table is linked to a specific world, rather than the entity ID space being linked to a specific world. I don't think she means "only let components be in one of the N possible worlds", we def want to let any component be used in any world.

So I think we are all on the same page?

[ElliottjPierce]

Oh I see. Yes, we're on the same page. Multiple storage systems per entity, each on different worlds/fragments.