Add a benchmark update policy.

Author: nnethercote

collector/benchmarks/README.md

@@ -246,3 +246,51 @@ Rust code being written today.
 - Compare the benchmarking time of the two versions.
 - When adding the new version, for `perf-config.json` and the `N-*.patch`
   files, use the corresponding files for the old version as a starting point.
+
+# Benchmark update policy
+
+rustc-perf is a "living benchmark suite" that is regularly changed. Some
+benchmarks in rustc-perf are verbatim copies of third-party crates. We
+periodically do a mass update of these benchmarks.
+
+Benefits of this approach:
+- We ensure we are measuring compilation of the crates most people are using.
+  This is most relevant for popular crates.
+- We get coverage of newer language features.
+
+Costs of this approach:
+- It takes time and effort.
+- We lose some data continuity.
+  - But the stable set of benchmarks used for the dashboard are not affected,
+    and they provide the greatest continuity.
+- If the code hasn't changed much, it won't have much effect.
+
+To balance these requires choosing a good refresh period. A period of three
+years is reasonable:
+- It's not too frequent (as 1 year would be, for example) while avoiding some
+  sub-optimal occurrences from the past (e.g. in 2022 numerous crates that were
+  more than five years old were updated).
+- It aligns with the Rust edition cycle.
+
+Update policy
+- The third-party crates should be updated in the year following a new edition.
+  (Based on the current edition cadence, this is every three years. If the
+  edition cadence changes in the future, this period should be reconsidered.)
+  The update should occur roughly six months after the edition hits a stable
+  Rust release, to give crate authors a decent amount of time to update to the
+  new edition.
+- All third-party crates that have had at least one new release should be
+  updated, even if not much code has changed. This avoids having to make
+  decisions about whether a crate has changed enough.
+- When doing this mass update, there may be some benchmarks that are deemed no
+  longer interesting and removed. Likewise, there may be some new benchmarks
+  that are added.
+- New versions should be added before old versions are removed, to ensure
+  continuity of profiling coverage.
+- The ad hoc addition and removal of individual benchmarks can continue
+  independently of this update cycle, as per the judgment of the rustc-perf
+  maintainers.
+
+History:
+- The first mass update of third-party crates occurred in [March/April
+  2022](https://hackmd.io/d9uE7qgtTWKDLivy0uoVQw).