Ordering of systems is hard: the problem and thoughts about the solution

[stepancheg]

Update

Barriers are implemented in #10618.

Original post

Hey,

I've been using bevy for a couple weeks, and here is my feedback about systems, which are either hard to use, are not explained well (which also means, hard to use). This is the part I struggled with the most.

It is entirely possible I don't understand ECS model well (which is new concept to me).

Also I didn't try to search very hard for similar discussions, so feel free to ignore this one.

So

Bevy is parallel. Which is great. But also non-deterministic, not great. For example:

• if a user controls a player with keyboard, reaction to the keyboard must happen in the same schedule iteration
• game are usually better when they are fully deterministic: if I open a save file, in 10 seconds the result should be the identical regardless of how many times I opened the save file

If I understood the documentation correctly, there are several main instruments to control ordering of systems:

• .after which is good because it allows ordering explicitly
• .in_set which allows ordering of group of systems relative to other group of systems
• .pipe which is nice because it allows passing data between system. Some kind of function call with dependency injection

Now the problem

Consider this system evaluation graph.

There are two ways to describe scheduling for this graph, both don't work well.

Either explicitly

a.before(d)
a.before(e)
b.before(d)
b.before(e)
b.before(f)
c.before(e)
c.before(f)

This is fragile and not scalable: you miss one edge, and then spend hours debugging why it doesn't work as expected on another machine with different number of CPUs where systems are executed in different order.

Another option is put (A, B, C) in one system set, and (D, E, F) in another system set

... and put one system set before another. This is not good, because it reduces parallelism: it adds dependency between A and F for example.

Additionally, this approach requires making everything public

For example, I have two modules (or two crates) defining some systems. I have to make all the functions and all the types public to make it work. Abstraction leaking, harder code navigation, slower compilation, etc.

How it could work

Resources

Resources are often initialized at startup, but also often consumed at startup.

Create a separate type like ResInit<FooBar>. So every system which consumes Res<FooBar> and ResMut<FooBar> must be schedule after any system which accepts ResInit<FooBar>.

For example:

fn init_res(a: ResInit<FooBar>) { ... }
fn init_more(a: Res<FooBar>) { ... }

app.add_system(Startup, (init_res, init_more))
// should be identical to
app.add_system(Startup, init_more.after(init_res))

Unwrap in resources

Additionally, would be nice if bevy tracked at graph construction time which resources are initialized. Basically:

• Res<T> should never be an "option", if you need it optional, make it Res<Option<T>>
• app should fail at graph construction time if there's a system with Res<T>, but no system with ResInit<T>

So if I forgot to add resource initialized, fail early with clear message instead of on some rare case during gameplay.

Events

If there are systems some are event readers, and others are event writers for the same event type, they should be ordered, so that:

• there's only one reader
• all writers are schedule before the reader
• fail at graph construction time if there's no writer

(Assuming these are in the same schedule, if they are not, it's fine as is.)

I.e.

fn produce_events(e: EventWriter<FooBar>) { ... }
fn produce_more_events(e: EventWriter<FooBar>) { ... }
fn consume_events(e: EventReader<FooBar>) { ... }

app.add(Update, (produce_events, produce_more_events, consume_events))
// should be identical too
app.add(Update, (produce_events.before(consume_events), produce_more_events.before(consume_events)))

Barrier

Add a new primitive, I'll call it barrier

// bevy
/// proc macro derived
trait Barrier { .. }

// These are markers to be consumed during graph construction, no data in them.
struct Before<B: Barrier>(...);
struct After<B: Barrier>(...);

// user code

#[derive(Barrier)]
struct CameraControl;

#[derive(Barrier)]
struct PlayerControl;

fn handle_keyboard(
  _: Before<PlayerControl>,
 _: Before<CameraControl>,
) { ... }

fn handle_mouse(
  _: Before<CameraControl>
} { ... }

fn update_player(
  _: After<PlayerControl>,
} { ... }

fn update_camera(
  _: After<CameraControl>,
) { ... }

app.add_system(
  Update,
  (handle_keyboard, handle_mouse),
)

// and in some other file

app.add_system(
  Update,
  (update_keyboard, update_mouse),
)

So the graph builder add before/after dependencies between these systems.

I think currently it is possible to kind of achieve similar behavior by creating a dummy empty system and schedule things before/after that dummy system at app initialization, but it is less explicit (system definition and system initialization are far apart in code, so it is fragile).

(Additionally, I'm scared of compiler optimizations. System identity seems to be a function pointer. And compiler at high optimization levels may collapse two identical functions into one, so if I use empty function as a barrier, something may not work as expected, but I didn't check.) I'm wrong, there's no identity.

Piping

.pipe is part I like, but not ideal:

• what if I need to produce a value for two systems?
• what if a system needs to consume outputs from two systems
• also, again it forces to leak abstractions: if I want some computation, I don't care who produced it

In my perfect view it would work like this:

/// Value that it produces by a system.
struct Produce<T>(pub T);

/// Value that is consumed by a system.
struct Consume<T>(pub T);

/// Value that can be consume by multiple systems.
struct ConsumeShared<'a, T>(pub &'a T);

How it would look like ideally, in my subjective opinion.

// Check the state and produce one time event.
fn check_game_over(...) -> Option<Produce<GameOver>> { ... }

// Triggered
// * only if `check_game_over` returned `Some`
// * on the same schedule iteration, if both systems are running on the same schedule
fn despawn_everything_on_game_over(game_over: ConsumeShared<GameOver>) { ... }

fn write_some_statistics_to_a_file(game_over: ConsumeShared<GameOver>) { ... }

I think this can probably be emulated with Changed filter. Don't know. But Changed filter has the same issue: it does not enforce that whoever is subscribed to change will be scheduled after the update was made.

That's it. Thanks for reading.

[Pieresqi]

I have encountered some of your pain points and learned to work around them. I will try to put my thoughts into points but it will be mess:

Resource initialization - I usually don't create system which initializes a resource. I impl FromWorld trait which gives me full access to World and from it I grab what I need. Then you can create the resource with init_resource<MyResource>() - This can be used outside systems or inside them with Commands if you want more granular initialization and I think resource which depend on each other need to be ordered but I am not sure. This way I do 99% of resource initialization and it works for me.

System dependencies - There are only few ways a systems depend on each other:

Events: for events I create a generic SystemSet - Producer and Consumer - which accepts any event and if I create in some plugin any event I also order it with configuration in that plugin and then I simply annotate systems with relevant set/event combinations - if you do this and you have system which consumes x event and produces y event then you should only annotate the consumer or you risk running into circular dependecy and stackoverflow later on, bevy has some detection for this but it's not perfect - this way events are consumed in the same frame...usually. I also do manual clearing of events and think multiple systems reading same event is anti pattern - there are exceptions to this like input which by it's nature has numerous different states and in different contexts mean different things.

Resources: I don't order systems against resources but against state - semantically resource existing at x time but not at x-y time means something "important" happened and it should be put into it's own state. If resource exists but it's fields are None then I don't think they should get their own state because both some and none are valid and equal. If you want to change resource to signify something so that some system can do something with it then you should use events - you are just abusing resources for message passing which is IMO anti pattern - yes events are resources but they are "specialized" for this and thus exception.

Entities/Components: because commands execution is postponed until apply_deffered then here it completely depends on you if want to wait a frame or state change or manually apply them.

States: if you systems do "contradictionary" things - load_chunks, unload_chunks - then you should use states and anotate systems with relevant state run conditions.

I think this cover majority of use cases of ordering - could be wrong.

I don't think before/after should be used outside of SystemSets because it tighly couples systems together and leaks systems to other plugins - use system set.

Also chain is very handy for ordering systems. You can nest chains, run condiitons and tuples to your heart's content.

Regarding your new primitive Barrier - to me it just looks like system set with different syntax.

I am not trying to rebuke your or something, this is just mine raving on this matter.

[stepancheg]

@Pieresqi thank you for the comment.

I did not understand all of it, but seems like you just simplify things by reducing concurrency. For example:

• by doing resource initialization using FromWorld, you initialize all resources in single thread
• by placing all events into generic producer/consumer, all consumers depend on the slowest producer, even if they don't need data every producer produces
• same issue with chain: it is safe-ish, but something that can be executed concurrently, isn't

And again, there's no protection against incorrect ordering of events.

[Pieresqi]

@stepancheg

by doing resource initialization using FromWorld, you initialize all resources in single thread

Does that matter ? If yoiu are doing it at startup then no. If you are using Commands then your systems will run in sequence with other Commands systems anyway.

by placing all events into generic producer/consumer, all consumers depend on the slowest producer, even if they don't need data every producer produces

You had the same idea but you are using before.
By it being generic then consumers X will depend on all producers X and don't care about producers Y and vice versa.

same issue with chain: it is safe-ish, but something that can be executed concurrently, isn't

Yeah, if you don't properly model system access in your head then it's easy to unnecessary restrict systems and reduce parallelism.

And again, there's no protection against incorrect ordering of events.

Yep, that would be nice.

[stepancheg]

For the record, I have implemented a PR for making dependencies between systems declarative: #10618.

Can be used like this:

fn first(_before: Before<MyBarrier>) {}
fn second(_after: After<MyBarrier>) {}

or like this with #[derive(SystemParam)] on structs wrapping EventReader/EventWriter and Before/After:

fn first(writer: MyOrderedEventWriter<Foo>) { ... }
fn second(reader: MyOrderedEventReader<Foo>) { ... }