|
2 | 2 | This directory contains various pieces of code for which we measure how fast do they execute
|
3 | 3 | when they are compiled with a specific version of `rustc`.
|
4 | 4 |
|
5 |
| -The benchmarks are located in crates that are part of the `runtime-benchmarks` workspace. Each crate |
6 |
| -contains a set of benchmarks defined using named closures. |
| 5 | +The benchmarks are located in several crates (`benchmark groups`) located in this directory. Each |
| 6 | +group defines a set of benchmarks using named closures. Each group should have a single, |
| 7 | +default, binary target, which will be executed by `collector`, and it should use the |
| 8 | +`run_benchmark_group` function from [`benchlib`](../benchlib) to define the benchmarks. |
7 | 9 |
|
8 |
| -Benchmarks are divided into sub-crates so that some benchmarks can use various versions of dependency |
9 |
| -crates and also so that they are grouped together by a relevant area (e.g. hashmap benchmarks). |
| 10 | +Runtime benchmarks are divided into groups so that some benchmarks can use different versions of |
| 11 | +dependency crates and also so that they are grouped together by a relevant area |
| 12 | +(e.g. hashmap benchmarks). |
| 13 | + |
| 14 | +## How are benchmarks executed |
| 15 | +The `collector` compiles each benchmark group and then invokes it with the `list` argument to list |
| 16 | +all benchmarks contained in the group. |
| 17 | + |
| 18 | +Then it executes each group with the `run` argument, which will cause `benchlib` to actually perform |
| 19 | +the benchmarks and output the results to `stdout` in JSON. That means that the benchmarked code should |
| 20 | +not print anything to `stdout`! |
| 21 | + |
| 22 | +Note that each benchmark group binary will be thus invoked twice per benchmarking collection. Keep this |
| 23 | +in mind so that the `main` function of the group doesn't perform too much work, which will be thrown |
| 24 | +away when it is invoked with the `list` argument. |
0 commit comments