@@ -249,6 +249,8 @@ Rust code being written today.
249
249
250
250
# Benchmark update policy
251
251
252
+ ## Background
253
+
252
254
rustc-perf is a "living benchmark suite" that is regularly changed. Some
253
255
benchmarks in rustc-perf are verbatim copies of third-party crates. We
254
256
periodically do a mass update of these benchmarks.
@@ -265,26 +267,19 @@ Costs of this approach:
265
267
and they provide the greatest continuity.
266
268
- If the code hasn't changed much, it won't have much effect.
267
269
268
- To balance these requires choosing a good refresh period. A period of three
269
- years is reasonable:
270
- - It's not too frequent (as 1 year would be, for example) while avoiding some
271
- sub-optimal occurrences from the past (e.g. in 2022 numerous crates that were
272
- more than five years old were updated).
273
- - It aligns with the Rust edition cycle.
270
+ ## Update policy
274
271
275
- Update policy
276
- - The third-party crates should be updated in the year following a new edition.
277
- (Based on the current edition cadence, this is every three years. If the
278
- edition cadence changes in the future, this period should be reconsidered.)
279
- The update should occur roughly six months after the edition hits a stable
280
- Rust release, to give crate authors a decent amount of time to update to the
281
- new edition.
272
+ - The third-party crates should be updated every three years. This is a
273
+ reasonable refresh period that is neither too short or too long. It happens
274
+ to match the Rust edition cycle, but this is just coincidence.
282
275
- All third-party crates that have had at least one new release should be
283
276
updated, even if not much code has changed. This avoids having to make
284
277
decisions about whether a crate has changed enough.
285
278
- When doing this mass update, there may be some benchmarks that are deemed no
286
- longer interesting and removed. Likewise, there may be some new benchmarks
287
- that are added.
279
+ longer interesting and removed. For example, in the 2022 update we found that
280
+ the `futures` crate was no longer interesting because all the functionality
281
+ had been split into sub-crates that rustc-perf doesn't measure. Likewise,
282
+ there may be some new benchmarks that are added.
288
283
- New versions should be added before old versions are removed, to ensure
289
284
continuity of profiling coverage.
290
285
- The ad hoc addition and removal of individual benchmarks can continue
0 commit comments