Skip to content

Commit e463d51

Browse files
committed
Address review comments.
1 parent ec77b28 commit e463d51

File tree

1 file changed

+10
-15
lines changed

1 file changed

+10
-15
lines changed

collector/benchmarks/README.md

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,8 @@ Rust code being written today.
249249
250250
# Benchmark update policy
251251
252+
## Background
253+
252254
rustc-perf is a "living benchmark suite" that is regularly changed. Some
253255
benchmarks in rustc-perf are verbatim copies of third-party crates. We
254256
periodically do a mass update of these benchmarks.
@@ -265,26 +267,19 @@ Costs of this approach:
265267
and they provide the greatest continuity.
266268
- If the code hasn't changed much, it won't have much effect.
267269
268-
To balance these requires choosing a good refresh period. A period of three
269-
years is reasonable:
270-
- It's not too frequent (as 1 year would be, for example) while avoiding some
271-
sub-optimal occurrences from the past (e.g. in 2022 numerous crates that were
272-
more than five years old were updated).
273-
- It aligns with the Rust edition cycle.
270+
## Update policy
274271
275-
Update policy
276-
- The third-party crates should be updated in the year following a new edition.
277-
(Based on the current edition cadence, this is every three years. If the
278-
edition cadence changes in the future, this period should be reconsidered.)
279-
The update should occur roughly six months after the edition hits a stable
280-
Rust release, to give crate authors a decent amount of time to update to the
281-
new edition.
272+
- The third-party crates should be updated every three years. This is a
273+
reasonable refresh period that is neither too short or too long. It happens
274+
to match the Rust edition cycle, but this is just coincidence.
282275
- All third-party crates that have had at least one new release should be
283276
updated, even if not much code has changed. This avoids having to make
284277
decisions about whether a crate has changed enough.
285278
- When doing this mass update, there may be some benchmarks that are deemed no
286-
longer interesting and removed. Likewise, there may be some new benchmarks
287-
that are added.
279+
longer interesting and removed. For example, in the 2022 update we found that
280+
the `futures` crate was no longer interesting because all the functionality
281+
had been split into sub-crates that rustc-perf doesn't measure. Likewise,
282+
there may be some new benchmarks that are added.
288283
- New versions should be added before old versions are removed, to ensure
289284
continuity of profiling coverage.
290285
- The ad hoc addition and removal of individual benchmarks can continue

0 commit comments

Comments
 (0)