Split benchmarks into primary and secondary categories

Kobzol · Kobzol · commit 02c29b568726 · 2022-03-07T23:15:46.000+01:00
diff --git a/collector/benchmarks/README.md b/collector/benchmarks/README.md
@@ -7,20 +7,31 @@ The suite changes over time. Sometimes the code for a benchmark is updated, in
 which case a small suffix will be added (starting with "-2", then "-3", and so
 on.)
 
-## Real programs that are important
+There are two categories of benchmarks, **Primary** and **Secondary**.
+
+## Primary
 
 These are real programs that are important in some way, and worth tracking.
+They mostly consist of real-world crates.
 
 - **cargo**: The Rust package manager. An important program in the Rust
   ecosystem.
 - **clap-rs**: A command line argument parser. A crate used by many Rust
   programs.
 - **cranelift-codegen**: The largest crate from a code generator. Used by
   Firefox.
+- **diesel**: A type safe SQL query builder. Utilizes the type system to
+  ensure a lot of invariants. Stresses anything related to resolving
+  trait bounds, by having a lot of trait impls for a large number of different
+  types.
+- **encoding**: Character encoding support. Contains some large tables.
 - **futures**: A futures implementation. Used by many Rust programs.
 - **helloworld**: A trivial program. Gives a lower bound on compile time.
+- **html5ever**: An HTML parser. Stresses macro parsing code significantly.
 - **hyper-2**: A fairly large crate. Utilizes async/await, and used by
   many Rust programs.
+- **inflate**: An old implementation of the DEFLATE algorithm. Stresses the
+  compiler in certain ways.
 - **piston-image**: A modular game engine. An interesting Rust program.
 - **regex**: A regular expression parser. Used by many Rust programs.
 - **ripgrep**: A line-oriented search tool. A widely-used utility.
@@ -33,40 +44,20 @@ These are real programs that are important in some way, and worth tracking.
   ecosystem.
 - **tokio-webpush-simple**: A simple web server built with tokio. Uses futures
   a lot.
-- **webrender**: A web renderer. Used by Firefox and Servo.
-- **webrender-wrench**: WebRender's test bench. An executable pulling in large
-  dependencies.
-
-## Real programs that stress the compiler
-
-These are real programs that are known to stress the compiler in interesting
-ways.
-
-- **diesel**: A type save SQL query builder. Utilizes the type system to
-  ensure a lot of invariants. Stresses anything related to resolving
-  trait bounds, by having a lot of trait impls for a large number of different
-  types.
-- **encoding**: Character encoding support. Contains some large tables.
-- **html5ever**: An HTML parser. Stresses macro parsing code significantly.
-- **inflate**: An old implementation of the DEFLATE algorithm. Stresses the
-  compiler in certain ways.
-- **keccak**: A cryptography algorithm. Contains a very high number of locals
-  and basic blocks.
 - **ucd**: A Unicode crate. Contains large statics that
   [stress](https://github.com/rust-lang/rust/issues/53643) the borrow checker's
   implementation of NLL.
 - **unicode_normalization**: Unicode character composition and decomposition
   utilities. Uses huge `match` statements that stress the compiler in unusual
   ways.
-- **wg-grammar**: A parser generator.
-  [Stresses](https://github.com/rust-lang/rust/issues/58178) the borrow
-  checker's implementation of NLL.
+- **webrender**: A web renderer. Used by Firefox and Servo.
+- **webrender-wrench**: WebRender's test bench. An executable pulling in large
+  dependencies.
 
-## Artificial stress tests
+## Secondary
 
-These are artificial programs that stress one particular aspect of the
-compiler. Some are entirely artificial, and some are extracted from real
-programs.
+These are either artificial programs or real crates that stress one particular aspect of the
+compiler in interesting ways.
 
 - **await-call-tree**: A tree of async fns that await each other, creating a
   large type composed of many repeated `impl Future` types. Such types caused
@@ -85,13 +76,17 @@ programs.
 - **deep-vector**: A test containing a single large vector of zeroes, which
   caused [poor performance](https://github.com/rust-lang/rust/issues/20936) in
   the past.
+- **derive**: A large amount of simple structs with a `#[derive]` attribute for common built-in traits such as Copy and Debug.
+- **externs**: A large amount of extern functions has caused [slowdowns in the past](https://github.com/rust-lang/rust/pull/78448).
 - **issue-46449**: A small program that caused [poor
   performance](https://github.com/rust-lang/rust/issues/46449) in the past.
 - **issue-58319**: A small program that caused [poor
   performance](https://github.com/rust-lang/rust/issues/58319) in the past.
 - **issue-88862**: A MCVE of a program that had a
   [severe performance regression](https://github.com/rust-lang/rust/issues/88862)
   when trying to normalize large opaque types with late-bound regions.
+- **keccak**: A cryptography algorithm. Contains a very high number of locals
+  and basic blocks.
 - **many-assoc-items**: Contains a struct with many associated items, which
   caused [quadratic behavior](https://github.com/rust-lang/rust/issues/68957)
   in the past.
@@ -103,7 +98,7 @@ programs.
 - **projection-caching**: A small program that causes extremely, deeply nested
   types which stress the trait system's projection cache. Removing that cache
   resulted in hours long compilations for some programs using futures,
-  actix-web and other libraries with similiarly nested type combinators.
+  actix-web and other libraries with similarly nested type combinators.
 - **regression-31157**: A small program that caused a [large performance
   regression](https://github.com/rust-lang/rust/issues/31157) from the past.
 - **token-stream-stress**: Constructs a long token stream much like the `quote`
@@ -122,5 +117,6 @@ programs.
 - **wf-projection-stress-65510**: A stress test which showcases [quadratic
   behavior](https://github.com/rust-lang/rust/issues/65510) (in the number of
   associated type bounds).
-- **externs**: A large amount of extern functions has caused [slowdowns in the past](https://github.com/rust-lang/rust/pull/78448).
-- **derive**: A large amount of simple structs with a `#[derive]` attribute for common built-in traits such as Copy and Debug.
+- **wg-grammar**: A parser generator.
+  [Stresses](https://github.com/rust-lang/rust/issues/58178) the borrow
+  checker's implementation of NLL.
diff --git a/collector/benchmarks/await-call-tree/perf-config.json b/collector/benchmarks/await-call-tree/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/cargo/perf-config.json b/collector/benchmarks/cargo/perf-config.json
@@ -1,5 +1,6 @@
 {
     "cargo_opts": "--lib",
     "runs": 1,
-    "touch_file": "src/cargo/lib.rs"
+    "touch_file": "src/cargo/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/clap-rs/perf-config.json b/collector/benchmarks/clap-rs/perf-config.json
@@ -0,0 +1,3 @@
+{
+  "category": "primary"
+}
diff --git a/collector/benchmarks/coercions/perf-config.json b/collector/benchmarks/coercions/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/cranelift-codegen/perf-config.json b/collector/benchmarks/cranelift-codegen/perf-config.json
@@ -1,4 +1,5 @@
 {
     "cargo_toml": "cranelift-codegen/Cargo.toml",
-    "touch_file": "cranelift-codegen/src/lib.rs"
+    "touch_file": "cranelift-codegen/src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/ctfe-stress-4/perf-config.json b/collector/benchmarks/ctfe-stress-4/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/deep-vector/perf-config.json b/collector/benchmarks/deep-vector/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/deeply-nested-async/perf-config.json b/collector/benchmarks/deeply-nested-async/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/deeply-nested-closures/perf-config.json b/collector/benchmarks/deeply-nested-closures/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/deeply-nested/perf-config.json b/collector/benchmarks/deeply-nested/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/derive/perf-config.json b/collector/benchmarks/derive/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/diesel/perf-config.json b/collector/benchmarks/diesel/perf-config.json
@@ -1,4 +1,5 @@
 {
     "cargo_toml": "diesel/Cargo.toml",	
-    "touch_file": "diesel/src/lib.rs"
+    "touch_file": "diesel/src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/encoding/perf-config.json b/collector/benchmarks/encoding/perf-config.json
@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/externs/perf-config.json b/collector/benchmarks/externs/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/futures/perf-config.json b/collector/benchmarks/futures/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "supports_stable": true
+    "supports_stable": true,
+    "category": "primary"
 }
diff --git a/collector/benchmarks/helloworld/perf-config.json b/collector/benchmarks/helloworld/perf-config.json
@@ -0,0 +1,3 @@
+{
+  "category": "primary"
+}
diff --git a/collector/benchmarks/html5ever/perf-config.json b/collector/benchmarks/html5ever/perf-config.json
@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/hyper-2/perf-config.json b/collector/benchmarks/hyper-2/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/inflate/perf-config.json b/collector/benchmarks/inflate/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "supports_stable": true
+    "supports_stable": true,
+    "category": "primary"
 }
diff --git a/collector/benchmarks/issue-46449/perf-config.json b/collector/benchmarks/issue-46449/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/issue-58319/perf-config.json b/collector/benchmarks/issue-58319/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/issue-88862/perf-config.json b/collector/benchmarks/issue-88862/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/keccak/perf-config.json b/collector/benchmarks/keccak/perf-config.json
@@ -0,0 +1,3 @@
+{
+  "category": "secondary"
+}
diff --git a/collector/benchmarks/many-assoc-items/perf-config.json b/collector/benchmarks/many-assoc-items/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/match-stress-enum/perf-config.json b/collector/benchmarks/match-stress-enum/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/match-stress-exhaustive_patterns/perf-config.json b/collector/benchmarks/match-stress-exhaustive_patterns/perf-config.json
@@ -1,4 +1,5 @@
 {
     "supports_stable": false,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "secondary"
 }
diff --git a/collector/benchmarks/piston-image/perf-config.json b/collector/benchmarks/piston-image/perf-config.json
@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "runs": 1
+    "runs": 1,
+    "category": "primary"
 }
diff --git a/collector/benchmarks/projection-caching/perf-config.json b/collector/benchmarks/projection-caching/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/regex/perf-config.json b/collector/benchmarks/regex/perf-config.json
@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/regression-31157/perf-config.json b/collector/benchmarks/regression-31157/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/ripgrep/perf-config.json b/collector/benchmarks/ripgrep/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "touch_file": "src/main.rs"
+    "touch_file": "src/main.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/serde/perf-config.json b/collector/benchmarks/serde/perf-config.json
@@ -1,4 +1,5 @@
 {
     "cargo_toml": "serde/Cargo.toml",
-    "touch_file": "serde/src/lib.rs"
+    "touch_file": "serde/src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/stm32f4/perf-config.json b/collector/benchmarks/stm32f4/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "cargo_opts": "--features=stm32f405"
+    "cargo_opts": "--features=stm32f405",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/style-servo/perf-config.json b/collector/benchmarks/style-servo/perf-config.json
@@ -4,5 +4,6 @@
     "cargo_toml": "components/style/Cargo.toml",
     "runs": 1,
     "supports_stable": true,
-    "touch_file": "components/style/lib.rs"
+    "touch_file": "components/style/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/syn/perf-config.json b/collector/benchmarks/syn/perf-config.json
@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/token-stream-stress/perf-config.json b/collector/benchmarks/token-stream-stress/perf-config.json
@@ -1,4 +1,5 @@
 {
     "cargo_opts": "--bin token-stream-stress-bin",
-    "touch_file": "src/main.rs"
+    "touch_file": "src/main.rs",
+    "category": "secondary"
 }
diff --git a/collector/benchmarks/tokio-webpush-simple/perf-config.json b/collector/benchmarks/tokio-webpush-simple/perf-config.json
@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/main.rs"
+    "touch_file": "src/main.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/tuple-stress/perf-config.json b/collector/benchmarks/tuple-stress/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "runs": 1
+    "runs": 1,
+    "category": "secondary"
 }
diff --git a/collector/benchmarks/ucd/perf-config.json b/collector/benchmarks/ucd/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "runs": 1
+    "runs": 1,
+    "category": "primary"
 }
diff --git a/collector/benchmarks/unicode_normalization/perf-config.json b/collector/benchmarks/unicode_normalization/perf-config.json
@@ -0,0 +1,3 @@
+{
+  "category": "primary"
+}
diff --git a/collector/benchmarks/unify-linearly/perf-config.json b/collector/benchmarks/unify-linearly/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/unused-warnings/perf-config.json b/collector/benchmarks/unused-warnings/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/webrender-wrench/perf-config.json b/collector/benchmarks/webrender-wrench/perf-config.json
@@ -1,5 +1,6 @@
 {
     "cargo_toml": "wrench/Cargo.toml",
     "runs": 1,
-    "touch_file": "wrench/src/main.rs"
+    "touch_file": "wrench/src/main.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/webrender/perf-config.json b/collector/benchmarks/webrender/perf-config.json
@@ -1,5 +1,6 @@
 {
     "cargo_toml": "webrender/Cargo.toml",
     "runs": 1,
-    "touch_file": "webrender/src/lib.rs"
+    "touch_file": "webrender/src/lib.rs",
+    "category": "primary"
 }
diff --git a/collector/benchmarks/wf-projection-stress-65510/perf-config.json b/collector/benchmarks/wf-projection-stress-65510/perf-config.json
@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}
diff --git a/collector/benchmarks/wg-grammar/perf-config.json b/collector/benchmarks/wg-grammar/perf-config.json
@@ -1,3 +1,4 @@
 {
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "secondary"
 }
diff --git a/collector/src/execute.rs b/collector/src/execute.rs
diff --git a/collector/src/main.rs b/collector/src/main.rs
diff --git a/database/src/bin/import-sqlite.rs b/database/src/bin/import-sqlite.rs
diff --git a/database/src/bin/postgres-to-sqlite.rs b/database/src/bin/postgres-to-sqlite.rs
diff --git a/database/src/bin/sqlite-to-postgres.rs b/database/src/bin/sqlite-to-postgres.rs
diff --git a/database/src/lib.rs b/database/src/lib.rs
diff --git a/database/src/pool.rs b/database/src/pool.rs
diff --git a/database/src/pool/postgres.rs b/database/src/pool/postgres.rs
diff --git a/database/src/pool/sqlite.rs b/database/src/pool/sqlite.rs
diff --git a/docs/glossary.md b/docs/glossary.md
diff --git a/site/src/api.rs b/site/src/api.rs
diff --git a/site/src/comparison.rs b/site/src/comparison.rs
diff --git a/site/static/compare.html b/site/static/compare.html

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+{`
	`2`	`+ "category": "secondary"`
	`3`	`+}`
Original file line number	Diff line number	Diff line change
`@@ -1,5 +1,6 @@`
`1`	`1`	`{`
`2`	`2`	`"cargo_opts": "--lib",`
`3`	`3`	`"runs": 1,`
`4`		`- "touch_file": "src/cargo/lib.rs"`
	`4`	`+ "touch_file": "src/cargo/lib.rs",`
	`5`	`+ "category": "primary"`
`5`	`6`	`}`
Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,5 @@`
`1`	`1`	`{`
`2`	`2`	`"cargo_toml": "cranelift-codegen/Cargo.toml",`
`3`		`- "touch_file": "cranelift-codegen/src/lib.rs"`
	`3`	`+ "touch_file": "cranelift-codegen/src/lib.rs",`
	`4`	`+ "category": "primary"`
`4`	`5`	`}`