minicrater: add some docs about it

Author: pietroalbini

README.md

@@ -44,4 +44,5 @@ guide](CONTRIBUTING.md).
 
 **Technical documentation:**
 
-* [Agent HTTP Api specification](docs/agent-http-api.md)
+* [minicrater docs](tests/minicrater/README.md)
+* [Agent HTTP API specification](docs/agent-http-api.md)

tests/minicrater/README.md

@@ -0,0 +1,59 @@
+# minicrater
+
+minicrater is the component of Crater's test suite that tests if the runs are
+actually executed and produce the correct result. It's inspired by rustc's
+compiletest.
+
+## Executing minicrater
+
+minicrater executions can take a few minutes to run, so it's ignored by default
+while running `cargo test`. You can run minicrater with:
+
+```
+$ cargo test minicrater -- --ignored --test-threads 1
+```
+
+The runs' output is hidden by default, but you can show it by setting the
+`MINICRATER_SHOW_OUTPUT` environment variable:
+
+```
+$ MINICRATER_SHOW_OUTPUT=1 cargo test minicrater -- --ignored --test-threads 1
+```
+
+## Adding tests to minicrater
+
+There are two ways to add a test to minicrater:
+
+* If your test requires different experiment configuration (like different
+  flags or `config.toml` file entries) and no other minicrater run has the
+  configuration you want you need to create a new minicrater run
+* Otherwise you can create a new local crate and have it tested by an existing
+  minicrater run (usually the full ones)
+
+minicrater runs are defined in `tests/minicrater/mod.rs`, and each run has
+additional configuration in the `tests/minicrater/<run>` directory: a
+`config.toml` with the configuration file used for the run, and
+`results.expected.json`, which contains the JSON output expected in the
+generated report.
+
+### Adding new local crates
+
+minicrater doesn't test public crates available on crates.io, but "local
+crates", which are dummy crates located in the `local-crates` directory at the
+top of the project. To add a new local crate you need to actually create the
+crate in that directory and then run:
+
+```
+$ cargo run create-lists
+```
+
+The `create-lists` command is only needed when you add or remove local crates,
+not when you edit one of them.
+
+### Adding new minicrater runs
+
+To add a new minicrater run, you need to add a new entry to
+`tests/minicrater/mod.rs` and a configuration file in
+`tests/minicrater/<run>/config.toml`. Then you run minicrater - expecting the
+experiment to be failing - and execute the command shown in the output to
+generate the expected JSON report.