First Class Multi-World Bevy

[ElliottjPierce]

TLDR

Other discussion posts have long requested Multi-World support in Bevy. From those discussions, the consensus seems to be that while multi-world Bevy is desirable, implementation is difficult. This discussion proposes a design for the feature, resolving a number of pain points in Bevy along the way. While this is relatively long, it is well worth the read in my opinion.

Background

Multi-World Bevy has been in discussion for a long time, and although the community is generally on board with it in theory, no consensus has been reached as to how to architect this feature. See discussions #7893 and #2237.

Further, this feature is hard to design at the moment because there are currently lots of ideas being thrown around regarding scheduling, pre-computing access lists for systems, systems as entities #16618, and lots of other considerations that would need to be figured out before a proper design for this can be made.

Motivation

Before we go further, I want to quickly clarify why multi-world bevy is so important to so many users.

In essence, many bevy worlds communicating together allows work to be divided and data to be categorized. This is an extremely useful feature that is available in most engines. For example, in Godot, this is a viewport with a custom World3d. Bevy's existing SubApp API comes close to this, except that SubApps can not be created to my knowledge after the main App starts. That is a sever limitation compared to other engines.

Here are a few examples of this feature in action:

Multiplayer

It is often nice to separate game logic and data from rendering and input logic and data for multiplayer games. Multi-world bevy would allow this by having a server world with the game logic, and a client world with the player/rendering logic. That way, a server just has to run the server world, a client just has to run the client world, and a listen server can just run both. Want to add split-screen? Just start another client world! Each client world could even communicate with different server worlds, allowing lots of flexibility. That's a bit of an over simplification. You also need to have replicated server worlds, but you get the idea. This pattern is extremely useful and is pretty strait forward in other engines, but after lots of experimenting, I think it's currently implausible for bevy.

Another advantage of this is the separation of data. For example, currently, we can't represent if a player is in their inventory or not as a bevy state because we could have multiple players. But, if each player is in their own world, this is trivial. As another example, let's say a local server world gets rolled back due to a bad prediction. The rollback here would have zero danger of corrupting data in player worlds. Currently, roll-back solutions have to tiptoe around non-game-logic entities and components to make sure the roll-back doesn't mess up input data for example. Multi-world bevy would eliminate a lot of bug potential here.

Editors and other tools

It would be nice to be able to inject an arbitrary world into an app that can freely communicate with it. For example, what if the bevy editor was just another world or group of worlds that went in a bevy app? Then, it can communicate directly with the main world instead of having to use bevy remote protocol. This would be much faster. This still probably won't be how the editor is made, but the concept of it makes sense as a feature.

Separate simulations

Take Minecraft for example. What if we could keep the nether and overworld loaded at all times? There would be no nether portal animation. Or, what if two player walk in opposite directions really far from the origin? We can't use floating point origin since they're in opposite directions. Right now, the only real option would be to use BigSpace, but in multi-world bevy, you could just through each player in their own client world. Tada! Problem solved.

I'll stop the examples here in the interest of brevity, but one could imagine lots of other uses for this too. Bevy for data science simulations? Multi-world support would help...

Existing Problems

As mentioned, Bevy already has API's that can be twisted to do some of this. The SubApp, Plugin, and World systems are especially relevant here. They all have problems, and multi-world bevy could fix all of them in my opinion.

Required Plugins

Sooo many plugins depend on others to work. Just look at Plugin. This is currently a pain point in my opinion. These dependencies should be enforced in the type system instead of documentation. It's a small detail, but it's one that can be confusing, especially for new users.

SubApp and App Overlap

This is a big one that causes lots of boiler plate. Look at the way events works! The same feature needs to be implemented for Apps, SubApps, and Worlds. (Though in this case the implementation on World is implicit.) Further, Plugins use almost exclusively the World API. Very few of them add SubApps or use set_runner. There's even a function SubApp::run_as_app which is telling. These APIs should be unified in some sense in my opinion.

SubApp extraction can only communicate between two Worlds

If I want one audio world to handle audio from multiple client worlds (for the sake of argument), how can I do that? I could make the source worlds sub apps of the audio world, but that's kind of backwards.

More generally, SubApp extraction represents a one to one relationship, but we sometimes want broader relationships.

Facilitating free flowing multi-world communication

I dealt with this a little in #16933. The short version is that SubApp extractions are not enough long term. We need systems that can reach into other world's data. For example, consider this system running on a player/client world:

fn display_player_inventory(inventory: OfServerWorld<Query<&InventorySlot>>) {
    ...
}

It's a bit contrived, but you get the idea. Client worlds need on demand, read-only access to server worlds. Ideally, this is configurable. For example, I should be able to declare that: Client world's update schedule needs read-only access to its server world.

Solutions

This proposes a new design for a Bevy App that, in my opinion, fixes the above problems. Please bare with me while I do my best to lay out the design. I want to again stress that this is just an opinion. If any version of this is ever implemented, it will probably look substantially different that what I explain here.

Lets go from the top down.

App

The basic idea here is Worlds as Entities. (When I first though of this idea, I laughed too.)

For now, just play along and see what I mean. Here's the basic structure:

pub struct App {
    root_world: World,
    runner: fn(Self) -> AppExit,
}

The root_world is the base World of the App. It stores SubApps as entities and can store other "one-per-App" structures. This is different from the main World. The main world is now a SubApp in the root_world. More on that in a bit.

The runner is the same idea as before. It would run update schedules on the root_world, etc.

SubApp

#[derive(Component, Deref, DerefMut)]
pub struct SubApp(pub World);

It's a component! To create a new SubApp/World, just add it to the root_world. Tada!

If you're thinking this is crazy, so did I. Hold that thought and watch this:

Plugin

pub trait Plugin: Component {
    fn build(sub_app: EntityMut);
}

Plugins are now simple components on SubApp entities. They can require the plugins/components they depend on, modify the SubApp they spawn on (as before), and they can hold relations. For example, a RenderWorld plugin and a Rendered plugin could use relations to link to each other's Worlds! More advanced features using hooks are also possible. In other words, SubApps::sub_apps is replaced with relations, just like the rest of bevy.

Modules

All of this is great, but how do we add functionality to the App itself? Modules. Modules are like the old plugins, and there can only be one per App.

pub trait Module: Resource {
    fn on_added(&mut self, root_world: &mut World);
    fn on_removed(&mut self,  root_world: &mut World);
}

They're resources! When modules are added, they get mutable access to the root_world. That means they can configure App schedules, etc. For example, a rendering Module could add a system to run the current rendering extraction phase by simply querying for entities with the RenderWorld plugin and and querying for entities with the Rendered plugin in relationship. In other words, we can reuse Bevy's powerful relations and other system to drive customizable extractions and communications between SubApps.

Further, since they are ultimately just resources, they can be added at runtime. This could facilitate modding and scripting in the future!

Solved Problems

Required Plugins

Plugins are Components now, so they can just mark the components they require. This is ultimately a better version of PluginGroups app.add_plugins(DefaultPlugins.set(WindowPlugin{ ...})) becomes root_world.spawn((DefaultPlugins, WindowPlugin{ ...})), just like the rest of bevy! Requirements > Bundles + PluginGroups.

SubApp and App Overlap

These are now fundamentally different concepts, so there's no more overlap. Instead of needing to Implement new functionality for App, SubApp, and World, you can just do so for World. If you need access to other plugins as well, you can use the EntityMut of a SubApp or some similar interface.

SubApp extraction can only communicate between two Worlds

Not anymore! Now, modules can use vanilla bevy systems to create custom extractions between as many SubApps as they like.

Facilitating free flowing multi-world communication

Ok, this design does not directly do this, not yet. As mentioned, we can't design this feature fully until we resolve ongoing improvements to scheduling and a few other issues discussed above. However, this design gets us to the point where root_world systems can gather any SubApps that need communication. So, when such a communication solution does exist, this design will allow it to be used much much more freely.

FAQ

I'll add to this section as more ideas are discussed.

How do we add Modules, SubApps, and Plugins after starting an app?

There are lots of ways to facilitate this, but the simplest would be to store a resource in SubApp's World with a CommandQueue for the root_world. We can build abstractions from there.

How does the root_world know when to end the App?

Well, if no SubApps are left, we can end the App. Determining when each SubApp closes can be done the same way it is now. SubApps can send close commands to the root_world which would despawn the SubApp. For SubApps with the WindowPlugin, this could be configured to fire automatically when the SubApp's main window is closed. In other words, we can reuse all the systems we have now for this at no cost while still supporting other SubApps, etc. We can even make a "close with" relationship to close render worlds, physics worlds, etc when their target SubApp is closed.

What if a Plugin is added that needs a Module that is not present?

The Module can be added when it spawns using hooks or some other abstraction. There are lots of options for this.

How can worlds communicate with each other without messing up access lists and the scheduler?

In the short term, we can just have a resource. For example, the render world may do something like:

#[derive(Resource)]
pub struct RenderedSourceWorld(*const World);

As long as the resource is inserted and removed during the lifetime of a reference to the RenderedSourceWorld, the pointer can be treated as a reference from the render world's perspective.

In the long term, we can build full abstractions here like I did in #16933. It did involve changing access lists, which is something we need to avoid for the sake of the schedulers. However, the scheduler ultimately just needs to know if two systems can run at the same time and on what threads; it doesn't need to know all the details of the access. In other words, if a system needs component A from world 1 and component B from world 2, we could make it so the access list just tells the scheduler, "I need components A and B". It can parallelize from there. Of course, we also need to ensure the system doesn't conflict with itself, but to do that, we could just create an access list for each world, sort of like treating each world as a system. If none of those systems conflict with itself, the multi-world system does not conflict with itself. Then, the access lists can just be merged together and proudly presented to the scheduler. At least, I think they can. May need a ECS-SME here.

How does this compare to SubWorlds

If you are not familiar. There is debate over if we should coordinate multiple worlds or divide one world into multiple partitions called SubWorlds. I am definitely more in favor of Multi-World Bevy. I have shared my thoughts on this below. I don't want to straw man SubWorld here either, so if there are considerations I'm missing, let me know.

Choosing Multi-World Bevy over SubWorlds

Most importantly, this proposal lets us reuse most of bevy's existing systems. If you only need one World, for like a scene or something, all you get is one World. This lets us keep the internals of a world and its storage as dense as possible. We can optimize the world as if it is the only thing that exists. With SubWorlds, there would be more considerations for even basic SystemParams. At the very least, each parameter would need to figure out which partition(s) its information comes from.

Also, Multi-World Bevy allows for multiple resources of the same type (one per world). This is really important for the player states in a multiplayer game. Of course that's just one example, but I already explained it above, so I won't list them further. It is worth mentioning that as resources become more entity-like, this may be possible in SubWorld architecture too, but again, that adds complexity. With multi-world bevy, we can keep the World API simple and fast, and add features on-top of it instead of inside of it.

Flexibility: I don't think I need to expound here much. A lot more is possible with a few worlds working together than one massive, possibly quite tangled, world.

Multi-World architecture requires us to create a system that manages connections between worlds. This proposal uses the new App and root_world for this, but there may be other solutions too. The reason I call this extra work a "pro" is because it allows us to re-use the world API for worlds themselves. As I've explained, this addresses required Plugins, relationships between groups of entities (Be they classic SubApps, other Worlds, or other SubWorlds/partitions), and requirements as a replacement for bundles/plugin groups.

Choosing SubWorlds over Multi-World Bevy SubWorlds

Well, SubWorlds is a bit simpler in some respects. We don't need to coordinate between worlds. But what that really means is that all that coordination work has moved to inside the World itself.

I don't think this would save us from any changes to the access list and scheduling system. Checking access between worlds is fundamentally the same problem as checking access between partitions.

One last Note

I know there are a lot ideas here. If you read the whole thing, thank you so much for your time.

Please keep in mind that these are just ideas at the moment. I want to put them out there because they propose a reasonable solution to a lot of problems and missing features I've seen discussed for a long time here. That is not to say this is the only path forward. I am leaving it up to core maintainers and other SMEs as to whether this is properly motivated and a good direction for Bevy.

As a result, I would appreciate keeping the discussion centered on use cases for this and pointing out flaws in the general idea, etc. We can discuss road maps and implementation details later, assuming core maintainers and the community like this approach.

I will be doing my best to reply to comments and concerns as they arise. If there are flaws with this idea, I'd much rather spot them early!

[maniwani]

Thanks for the write up!

The issue I've always had with "multi-world" is it eventually wants to clash with the access model, i.e.

We need systems that can reach into another world's data.

For various reasons, I'm doubtful that'll work out. I think it'd be wiser for the community to focus on addressing whatever is insufficient about a single world.

Looking at the motivations

Rendering

It wasn't mentioned, but IIRC pipelined rendering kickstarted the use of multiple worlds in Bevy. AFAIK the idea of cloning entities to a second world was inspired by this GDC talk.

Rendering SMEs may disagree with me, but in hindsight I think "clone entities into a second world" was a mistake. It was/is the root of a lot of technical debt. Bevy could still do pipelined rendering without a second world. The second world just enables an earlier sync point. On paper this means more of the frame N+1 simulation work and frame N rendering work can overlap (which could mean higher FPS), but I think that's very unlikely to see in practice.

I think a stronger architecture would scrap the render world, upload data straight from the main world to the GPU, and do GPU-driven rendering. (The render thread can still be there to submit the GPU command queue so that the main thread isn't blocked from beginning the next frame.)

Griffin shared an experiment (see this Discord discussion) with a setup like that and IIRC it was much faster than the 0.15 renderer.

Multiplayer

Paraphrasing what you said a little:

Currently, multiplayer plugins have to tiptoe around non-replicated entities and components to make sure that rolling state back doesn't mess things up.

You're right. It's challenging to restore a world to an earlier state because of the way networked and non-networked data is mixed together inside it.

To expand on why (so that hopefully the stuff I'm gonna suggest further down makes sense), if you add or remove components from one group, it'll displace components from the other group because archetypes and tables in the world don't keep them separate. This is also why you can't easily save and rollback "all networked components" (you instead have to visit those entities one by one).

The other thing is all clients need agreed-upon IDs for networked entities and components, but Entity is currently too opaque to serve that role. This is because all entities in the world, networked or not, get their Entity IDs from the same pool.

Scene Management

Based on the example you gave for "separate simulations", I think it may be more apt to call it a scene management issue?

IIUC you're talking about situations where you'd like several groups of entities to be present in the World at the same time, but only want queries and systems to see a subset of them.

That usecase is really interesting. In lieu of greater querying capabilities (like flecs has), I'm inclined to agree that multiple worlds might be the only workable solution for it (provided it's easy to install the same systems in those worlds).

(Queries in flecs, for example, can find matching archetypes by traversing relationships. So like, a query can literally require "entity is descendant of <current scene root>" as a term.)

How single world UX can be improved

Based on the above, what World is missing are methods of partitioning Entity allocation and component storage, and also (minimally disruptive) methods for cloning and restoring partitioned component data.

Partitioning Entity allocation (through reservation)

To simplify replicating entities, it'd be best if users could just reserve specific entity indices from the World. Like, completely disconnect them from the general freelist (to form or append to a private freelist).

Users needing to map Entity to a different ID always seemed fragile to me considering components can store Entity values.

Partitioning component storage

Multi-world has been proposed as a means of partitioning components (e.g. into networked and non-networked).

My suggested alternative is, instead of using multiple worlds as partitions, introduce partitioning inside World, i.e. allow component types to be (re)assigned to different partitions, with each partition having its own archetypes and tables.

A structure-preserving Clone

It is possible to clone a world (or partition) in a way that every table and archetype in the original gets an exact counterpart in its clone, whether or not all the components actually implement Clone.

If an archetype/table has !Clone components, just substitute those with zero-sized markers/tags (can take the original ComponentId of T and toggle a flag bit in it to create a ComponentId for "placeholder for T").

This should be much faster than cloning entities one by one.

Bonus: Yeet SubApp

I think Bevy would benefit from having fewer layers of abstractions. If Bevy would defer building plugins in order to build them in topological order, SubApp wouldn't be necessary. Any secondary world could be spawned and temporarily (or permanently) stored as an entity in the main world.

At that point, I think each plugin should be its own entity / slightly-fancier-than-normal exclusive system.

pub trait Plugin {
    /// A system that installs the "content" of the plugin. This is where the user would typically register the
    /// plugin's components and schedule its systems.
    fn build(world: &mut World);
}

And App could look like this to encourage users to do all their configuration through the ECS.

pub trait App {
    /// Prepares an environment to run the application.
    /// 
    /// The (IMO ideal) default implementation of this function will spawn a `winit` event loop, spawn a thread
    /// and a [`World`] inside it, and run [`setup`].
    fn run() {
        /* default implementation */
    }

    /// A system that acts on the world before any others. This is where the user would typically
    /// import plugins, etc. 
    fn setup(world: &mut World);
}

[maniwani]

I am not proposing that we make queries that can pull from multiple worlds. I am suggesting that we can let systems have parameters that pull from one and only one world each.

I am definitely not suggesting we re-build querying to support something like: Query<FromServerWorld<&Backpack>, With<OpenBackpack>>.

Well, your example resembles how the renderer uses the Extract system param, but more refined (formally storing worlds as entities and choosing one to borrow from, etc.)

Given what you said here, I guess I was overreacting before. Pursuing a more thorough multi-world API wouldn't mean never introducing a form of internal partitioning, so it seems like an OK direction overall.

But yeah, partitioning can be useful on its own, not just in service of parallelism, so I think it's just good to be aware how multiple worlds could be a little awkward if someone isn't doing it for parallelism.

[maniwani]

One last hesitancy I have for subwords is this situation: What if I want multiple partitions that have the same purpose? For example, simulation islands for physics, or running two clients for split-screen, or any number of other situations. As long as we can create partitions at runtime, and specify them uniquely by a runtime id instead of type, I would be happy with this. Partitions as Entities?

Hmm, I haven't really thought about those cases. I focused on subworld-style partitioning mostly as a way to handle rollback without losing the ability for one query to get all of an entity's components (I might be putting too much importance on this).

🤔 I'm not sure if I would create partitions for split screen. My first thought is to put both players under the same client app. Locally, another player is just one more set of inputs to process and one more camera view to render and composite.

For physics islands and other scenarios, I'm not sure. I do think runtime creation would be doable, but the way I imagine sub-world partitioning being implemented does nothing for parallelism (my concept of it just changes how component storage is fragmented). If you did want true parallelism, I think you would genuinely need multiple worlds.

tbh I feel like my opinions on component partitioning / sub-worlds are probably my hottest take. The entity ID reservation and structure-preserving clone stuff would be just as useful with multiple worlds.

[ElliottjPierce]

Ok that gives some more clarity.

I'm not sure if I would create partitions for split screen.

I agree. But, that's the only way I can think of to use bevy's states for players. States like IsInInventory for example could use state transition schedules and observers to have much cleaner, more responsive code than having to use events or something, where each event passes a player id and a new state of some kind.

Partitions for each client also opens up capabilities like floating origin and things like different texture packs for different clients. Honestly this example is a bit contrived, but it makes sense to have logically separate partitions that have the same properties. This might even be nice for speed because the groups could be more dense to iterate, especially if depending on which group/partition an entity is in, a different region of memory is loaded. Pushing all those conditions together could decrease cache misses. And in multi-world bevy, to your point, this could also be parallelized.

I do think runtime creation would be doable

Phew! That's really important IMO.

If you did want true parallelism, I think you would genuinely need multiple worlds.

In my opinion, parallelism shouldn't be a consideration here at all. Even in multi-world bevy, a perfect scheduler would still be able to parallelize more with one big schedule than by running each partition/world as its own thread. Precomputed schedulers are on the distant horizon I think, so parallelism itself isn't a real motivation for me.

Rather, I think the benefit of partitions/multiple worlds is being able to group entities by partition and fragment storage to keep similar entities close together. It think this could reduce cache misses some, but more importantly, it gives physical separation to a logical difference. For example, a rollback can't possibly mess up input data if the input data is in a fundamentally different place (like another world or partition). Of course, this assumes we can limit the scope of a system to only a subset of partitions/worlds. This is trivial/emergent with multi world, but IDK about partitions.

I feel like my opinions on component partitioning / sub-worlds are probably my hottest take.

Blazing hot. But definitely not wrong. I actually really like the idea if we can facilitate the features I've mentioned. Ultimately, if we do find a way to parallelize SubWorlds anyway (which is not the immediate goal), it would still be faster than parallel multi-world because it would save the entity translation/query joining. That would be cool.

The entity ID reservation and structure-preserving clone stuff would be just as useful with multiple worlds.

I agree. I would like to come away from this discussion with some form of a loose roadmap for either multi-world bevy or SubWorlds, but in the meantime, since this would benefit both, it might be worth looking into in the short term too.

[Atlas16A]

For various reasons, I'm doubtful that'll work out. I think it'd be wiser for the community to focus on addressing whatever is insufficient about a single world.

Ultimately the thing that single world is insufficient for is "There is no concrete way to isolate data from certain things."
If I have entities I don't want a external system accessing, ie editor ui vs game systems, there is absolutely zero way for me to tell bevy "prevent queries from gathering this entity unless explicitly asked for."

[ElliottjPierce]

there is absolutely zero way for me to tell bevy "prevent queries from gathering this entity unless explicitly asked for."

That is exactly right. That is the chief motivation for me about multi-world bevy, but it is not the only motivation.

Also think about schedules. If we want to continue to split each update into multiple schedules, it might be beneficial for disjoint schedules to run at the same time. For example, the render world may want to run it's schedules concurrently with the main world's schedules. Merging this into the same world makes that impractical.

Strictly speaking, bevy already has multi-world support. It's just in the form of SubApps, which are painful, overly restrictive and boiler-plate heavy, and come with a lot of limitations. I'm just proposing a way to improve the multi-world experience.

Many (but not quite all) of these motivations are possible in other ways. Ex. default query filters, entity/component partitions, etc. But for now, Bevy has chosen SubApps/Multiple worlds; it's just that the current implementation of that is really clunky.