Shouldn't the examples include tests?

[Voultapher]

I talked with someone about new age game engines getting a better grip on mutable state and the effects that should have on being able to write automated tests for games. However searching for the word test in the examples/ directory showed practically no testing. Shouldn't the project provide some documentation, I couldn't find a mention of writing tests in the bevy book, and examples how to test games written with bevy?

[NiklasEi]

There is an example for testing systems in the repository: https://github.com/bevyengine/bevy/blob/main/tests/how_to_test_systems.rs

But I agree that we could add at least one example that demonstrates how one would add tests to a minimal game.

[alice-i-cecile]

This is a pretty reasonable take :) There are a number of distinct problems here that could be addressed separately:

1. Our examples themselves don't have tests. This would also be useful for better assurances from our CI.
2. The example linked by @NiklasEi isn't discoverable enough.
3. The integration testing story is, in general, still frustratingly weak (see Allow bevy to be run as an integration test inside a #[test] #1057).
4. Unit testing groups of systems is often heavy on boilerplate, and hard to integrate well.
5. We don't have great paradigms for automated testing of graphics.

[cart]

1. Examples should not have tests unless the purpose of the example is to show how to write tests. Examples are intentionally minimal: they should include exactly as much code as needed to illustrate one thing, and nothing else.
2. Imo a top level tests folder is the most discoverable we can make test examples. The only improvement I can think of here is adding a testing section to the bevy book.
3. We've merged pull requests that resolve the issues in Allow bevy to be run as an integration test inside a #[test] #1057: Add option to ignore events when receving unknown WindowId #1072, Add an alternative winit runner that can be started when not on the main thread #1063, Prevent double panic in the Drop of TaksPoolInner #1064
4. Any easy examples you can point to that illustrate this boilerplate?
5. Truuuuth. Definitely lots of work to do here.

[alice-i-cecile]

1. Fair, agreed.
2. IMO a examples/testing folder would be more discoverable. Top level tests folder feels like it should be for tests of the engine itself.
3. Oh wow, that's very exciting. I will investigate and update the issue with my findings!
4. a) synchronizing systems used in tests vs app code is challenging, especially without the ability to use App and plugins b) Ergonomic tools for working with Commands from &mut World #3096 is really rough when testing commands c) spawning complex test scenes using only-code is rough d) generally, the pattern I want to use is to run a subset of my App for one or more frames, then assert that a system returns a valid result when evaluated on the world. All three parts of that are pretty heavy to handle.
5. We'll get there!

[mockersf]

about 2.:

• tests are mentioned in the README: https://github.com/bevyengine/bevy/tree/main/examples#tests
• we could add an empty folder tests in examples that would link back to the root tests folder
• tests in examples are not run, so they would be dead code we could add a cargo test --examples in CI to run tests in examples, I didn't know that worked