Benchmarking Rayon

Benchmarking Rayon is very much a "work in progress". It is hard to estimate the impact of a change to the core scheduler across the wide diversity of machines and workloads that people encounter in practice. Also, we don't have a wide set of benchmarks showing representative use cases of Rayon "in production". So, if you are using Rayon in production, contributing benchmarks would be very helpful!

With that said, this page exists to try and document some of the benchmarks and tooling considerations we do have.

Document your setup

It may seem obvious, but the first thing when reporting any results is to document your setup. What OS are you running? How many cores do you have? And so forth. It's hard to do a thorough job -- even bash environment variables can affect performance -- but it's important to have at least a basic picture.

Testing rayon-demo

The most basic thing to run is rayon-demo. I often use the cargo-chrono tool to test the impact of multiple commits. The tool allows you to do multiple runs and compute the median, which helps to eliminate the impact of noise. The following is a standard command line of mine. You have to change the XXX as appropriate to name the PR and the branch you are testing:

cargo-chrono bench -f rayon-prXXX.csv --commits "nikomatsakis/master XXX" --repeat 15 \
    --ignore-dirty '*pr*csv' --ignore-dirty '*svg' \
    join_microbench \
    nbody_par \
    quick_sort_par_bench \
    merge_sort_par_bench \
    parallel_find_last \
    parallel_find_missing \
    dj10 \
    fibonacci_split_recursive

This will generate a file (rayon-prXXX.csv) containing the measurement results. You can then generate plots like so:

cargo-chrono plot -f rayon-prXXX.csv --output-file rayon-prXXX-points.svg
cargo-chrono plot -f rayon-prXXX.csv --output-file rayon-prXXX-points-medians.svg --medians --normalize

The first command shows all measurements. The second command computes medians, normalized to 100. This is pretty useful for detecting deviations one way or the other. I usually eyeball the first plot to make sure the measurements are clustered in a tight little group and then look at the second one to see if anything really changed.

Testing servo

Servo is a big application that is now using rayon in a number of different ways.

XXX this needs to be written