From 7641de1ba91ab75369c4b0c616032e84959165df Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Wed, 7 Oct 2020 23:00:47 -0700 Subject: [PATCH 1/8] Initial draft of `doctest-name` RFC This RFC proposes the ability to annotate documentation test code blocks with metadata of the form `name=NAME`. The given name will be used to identify the documentation test when it is run by the test runner, in addition to the current information of test binary, module, and line number. For example, this documention test: ```name=arithmetic assert_eq!(2 + 2, 4); ``` Assuming that it is in crate `foo`, in file `src/lib.rs`, in module `bar`, on line 37, will produce this output when run: Doc-tests foo running 1 test test src/lib.rs - bar::arithmetic (line 37) ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out --- text/0000-doctest-name.md | 142 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 text/0000-doctest-name.md diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md new file mode 100644 index 00000000000..23ce2cedb07 --- /dev/null +++ b/text/0000-doctest-name.md @@ -0,0 +1,142 @@ +- Feature Name: doctest-name +- Start Date: 2010-10-07 +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) + +# Summary +[summary]: #summary + +Allow giving documentation tests names for identification in teest runner +output by including `name=NAME` in doctest code block info strings. + +# Motivation +[motivation]: #motivation + +When doctests run, the test runner prints the name of the module that contains +them and the line number that they start on. If a module contains many +doctests, it is hard to identify which doctest is which in the test runner +output by line number alone. + +By giving users the option to give names to doctests, identifying individual +doctests in test runner output will be easier. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +Doctests are written as fenced markdown code blocks that start and end +with lines containing three backticks. + +The first set of backticks may be followed by an info string. For example, to +tell the Rust compiler to ignore the doctest: + +````rust + +// This doctest won't run. +```ignore +assert_eq!(2 + 2, 4); +``` + +```` + +Normally, doctests do not have names, and when run, the test runner will +identify them by their test binary, module name, and line number: + +``` + Doc-tests foo + +running 1 test +test src/lib.rs - bar (line 37) ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out +``` + +If you'd like to have the test runner print a more informative name, perhaps +because there are many doctests in a single file, you can put `name=NAME`, +where `NAME` is the name the test should have, in the doctest's info string. +`NAME` should be a valid Rust identifer. + + +````rust + +```name=arithmetic +assert_eq!(2 + 2, 4); +``` + +```` + +``` + Doc-tests foo + +running 1 test +test src/lib.rs - bar::arithmetic (line 37) ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out +``` + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +The set of valid words that may appear in code block info strings is extended +to include patterns of the form `name=IDENT`. + +This `IDENT` will be the name of the documentation test generated by the code +block. + +When the test runner runs the documentation test, the test will be identified +by this name, in addition to the test binary, module, and line number. + +Multiple `name=IDENT` words may not appear in a single code block's info +string. + +`name=IDENT` may be combined with existing annotations like `rust` or +`should_panic` by separating the annotations with commas: + +```` + +```rust,name=foo +assert!(true) +``` + +```` + +The ability to include multiple comma-separated annotations is an existing +feature. + +# Drawbacks +[drawbacks]: #drawbacks + +- Adds complexity to code block parsing. + +- Requires parsing multi-word info strings, which are standard markdown, but + which does increase the complexity of the markdown parser. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +This design seems like the simplest design possible, although viable +alternatives may have been overlooked. + +A previous version of this RFC included the additional restrictions that labels +must be valid rust identifiers, and that labels must be unique within the same +module. These restrictions were intended to allow the machinery that generates +doctest tests to use the labels as the generated doctest function names. If +this simplifies or improves the generated doctest functions, these restrictions +might be desirable. + +The impact of adopting this RFC is minimal. Doctests test output will remain a +little hard to make sense of. + +# Prior art +[prior-art]: #prior-art + +None. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +None. + +# Future possibilities +[future-possibilities]: #future-possibilities + +None. From 78f4c1d2c9bc1c5fd172c3cc635e41f775527165 Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Thu, 8 Sep 2022 10:03:54 -0700 Subject: [PATCH 2/8] Typo: teest -> test Co-authored-by: Aleksey Kladov --- text/0000-doctest-name.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md index 23ce2cedb07..574e4071ad8 100644 --- a/text/0000-doctest-name.md +++ b/text/0000-doctest-name.md @@ -6,7 +6,7 @@ # Summary [summary]: #summary -Allow giving documentation tests names for identification in teest runner +Allow giving documentation tests names for identification in test runner output by including `name=NAME` in doctest code block info strings. # Motivation From 1ccd36eca4a0f0d3b7eae2898b4b7e2e28331693 Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Wed, 21 Sep 2022 21:24:31 -0700 Subject: [PATCH 3/8] Address feedback - Mention test filtering - Remove restriction that `NAME` be a valid identifier - Rename IDENT to NAME --- text/0000-doctest-name.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md index 574e4071ad8..78b1ec54bb0 100644 --- a/text/0000-doctest-name.md +++ b/text/0000-doctest-name.md @@ -18,7 +18,8 @@ doctests, it is hard to identify which doctest is which in the test runner output by line number alone. By giving users the option to give names to doctests, identifying individual -doctests in test runner output will be easier. +doctests in test runner output will be easier. Additionally, users could re-run +specific tests using `cargo test --doc -- $TEST_NAME`. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation @@ -53,7 +54,6 @@ test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out If you'd like to have the test runner print a more informative name, perhaps because there are many doctests in a single file, you can put `name=NAME`, where `NAME` is the name the test should have, in the doctest's info string. -`NAME` should be a valid Rust identifer. ````rust @@ -77,18 +77,18 @@ test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out [reference-level-explanation]: #reference-level-explanation The set of valid words that may appear in code block info strings is extended -to include patterns of the form `name=IDENT`. +to include patterns of the form `name=NAME`. -This `IDENT` will be the name of the documentation test generated by the code +This `NAME` will be the name of the documentation test generated by the code block. When the test runner runs the documentation test, the test will be identified by this name, in addition to the test binary, module, and line number. -Multiple `name=IDENT` words may not appear in a single code block's info +Multiple `name=NAME` words may not appear in a single code block's info string. -`name=IDENT` may be combined with existing annotations like `rust` or +`name=NAME` may be combined with existing annotations like `rust` or `should_panic` by separating the annotations with commas: ```` @@ -129,7 +129,8 @@ little hard to make sense of. # Prior art [prior-art]: #prior-art -None. +Unit and integration tests have names. This change would bring doctests closer +to parity, by allowing features like test filtering to work with doctests. # Unresolved questions [unresolved-questions]: #unresolved-questions From 9472e1a2175b31bf7355b308de0a42d36daf8938 Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Tue, 24 Jan 2023 21:10:48 -0800 Subject: [PATCH 4/8] Require that test names be valid rust identifiers --- text/0000-doctest-name.md | 30 ++++++++++++------------------ 1 file changed, 12 insertions(+), 18 deletions(-) diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md index 78b1ec54bb0..5f290f75f81 100644 --- a/text/0000-doctest-name.md +++ b/text/0000-doctest-name.md @@ -6,8 +6,8 @@ # Summary [summary]: #summary -Allow giving documentation tests names for identification in test runner -output by including `name=NAME` in doctest code block info strings. +Allow giving documentation tests names for identification in test runner output +by including `name=IDENT` in doctest code block info strings. # Motivation [motivation]: #motivation @@ -24,8 +24,8 @@ specific tests using `cargo test --doc -- $TEST_NAME`. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -Doctests are written as fenced markdown code blocks that start and end -with lines containing three backticks. +Doctests are written as fenced markdown code blocks that start and end with +lines containing three backticks. The first set of backticks may be followed by an info string. For example, to tell the Rust compiler to ignore the doctest: @@ -52,8 +52,9 @@ test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` If you'd like to have the test runner print a more informative name, perhaps -because there are many doctests in a single file, you can put `name=NAME`, -where `NAME` is the name the test should have, in the doctest's info string. +because there are many doctests in a single file, you can put `name=IDENT`, +where `IDENT` is the name the test should have, which must be an identifier, in +the doctest's info string. ````rust @@ -77,18 +78,18 @@ test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out [reference-level-explanation]: #reference-level-explanation The set of valid words that may appear in code block info strings is extended -to include patterns of the form `name=NAME`. +to include patterns of the form `name=IDENT`. -This `NAME` will be the name of the documentation test generated by the code -block. +This `IDENT` will be the name of the documentation test generated by the code +block, and must be a valid Rust identifier. When the test runner runs the documentation test, the test will be identified by this name, in addition to the test binary, module, and line number. -Multiple `name=NAME` words may not appear in a single code block's info +Multiple `name=IDENT` words may not appear in a single code block's info string. -`name=NAME` may be combined with existing annotations like `rust` or +`name=IDENT` may be combined with existing annotations like `rust` or `should_panic` by separating the annotations with commas: ```` @@ -116,13 +117,6 @@ feature. This design seems like the simplest design possible, although viable alternatives may have been overlooked. -A previous version of this RFC included the additional restrictions that labels -must be valid rust identifiers, and that labels must be unique within the same -module. These restrictions were intended to allow the machinery that generates -doctest tests to use the labels as the generated doctest function names. If -this simplifies or improves the generated doctest functions, these restrictions -might be desirable. - The impact of adopting this RFC is minimal. Doctests test output will remain a little hard to make sense of. From cc3c07a46e8f330f4d843ee6c6ece24f544649eb Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Tue, 24 Jan 2023 21:21:16 -0800 Subject: [PATCH 5/8] Use spaces instead of commas --- text/0000-doctest-name.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md index 5f290f75f81..73269232649 100644 --- a/text/0000-doctest-name.md +++ b/text/0000-doctest-name.md @@ -90,11 +90,11 @@ Multiple `name=IDENT` words may not appear in a single code block's info string. `name=IDENT` may be combined with existing annotations like `rust` or -`should_panic` by separating the annotations with commas: +`should_panic` by separating the annotations with spaces: ```` -```rust,name=foo +```rust name=foo assert!(true) ``` From 424f3cd87baa675b69ee6a7199d88e1723db3bce Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Tue, 14 May 2024 17:37:35 -0700 Subject: [PATCH 6/8] Use commas to separate attributes --- text/0000-doctest-name.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md index 73269232649..5f290f75f81 100644 --- a/text/0000-doctest-name.md +++ b/text/0000-doctest-name.md @@ -90,11 +90,11 @@ Multiple `name=IDENT` words may not appear in a single code block's info string. `name=IDENT` may be combined with existing annotations like `rust` or -`should_panic` by separating the annotations with spaces: +`should_panic` by separating the annotations with commas: ```` -```rust name=foo +```rust,name=foo assert!(true) ``` From 7f90cc4781a421d5bc5a26b486c1909a593b8602 Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Sun, 19 May 2024 03:11:10 -0700 Subject: [PATCH 7/8] Note that raw identifiers are permitted --- text/0000-doctest-name.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md index 5f290f75f81..a4298c35453 100644 --- a/text/0000-doctest-name.md +++ b/text/0000-doctest-name.md @@ -81,7 +81,8 @@ The set of valid words that may appear in code block info strings is extended to include patterns of the form `name=IDENT`. This `IDENT` will be the name of the documentation test generated by the code -block, and must be a valid Rust identifier. +block, and must be a valid Rust identifier, which includes raw `r#`-prefixed +identifiers. When the test runner runs the documentation test, the test will be identified by this name, in addition to the test binary, module, and line number. From 8d13ad0059396cf9ff35b43d1c974fd816c4da7c Mon Sep 17 00:00:00 2001 From: Casey Rodarmor Date: Mon, 20 May 2024 13:54:12 -0700 Subject: [PATCH 8/8] Warn if IDENTs are non-unique. --- text/0000-doctest-name.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0000-doctest-name.md b/text/0000-doctest-name.md index a4298c35453..ed26c5f0aba 100644 --- a/text/0000-doctest-name.md +++ b/text/0000-doctest-name.md @@ -90,6 +90,10 @@ by this name, in addition to the test binary, module, and line number. Multiple `name=IDENT` words may not appear in a single code block's info string. +Non-unique `IDENT`s and `IDENTS` that conflict with other items within the +current namespace will produce a warn-by-default error that may become a hard +error in the future. + `name=IDENT` may be combined with existing annotations like `rust` or `should_panic` by separating the annotations with commas: