Skip to content

Commit 06e8440

Browse files
calebcartwrightepage
authored andcommitted
docs: elaborate on a few points
1 parent ce86749 commit 06e8440

File tree

1 file changed

+50
-15
lines changed

1 file changed

+50
-15
lines changed

text/0000-t-test.md

Lines changed: 50 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,71 +1,106 @@
11
- Feature Name: n/a
2-
- Start Date: (2023-06-07)
2+
- Start Date: (2023-06-27)
33
- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000)
44
- Rust Issue: n/a
55

66
# Summary
77
[summary]: #summary
88

9-
Create a team focused on the testing in the development workflow under primarily under [t-devtools](https://www.rust-lang.org/governance/teams/dev-tools) but also working closely with [t-libs](https://www.rust-lang.org/governance/teams/library) when it comes to the API and implementation of the built-in test harness.
9+
Create a new subteam focused on testing in the development workflow and to be responsible for
10+
ensuring Rust has a high quality automated testing experience that includes the capabilities
11+
developers expect when working with a modern programming language.
1012

1113
# Motivation
1214
[motivation]: #motivation
1315

14-
In [rust 1.70.0](https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html), a bug was fixed so unstable test features, like `cargo test -- --format json`, require using a nightly toolchain, like other unstable features. The problem is IDEs and CIs have been relying on this behavior to get programmatic test data and little progress has been made in the last 5 years ([#49359](https://github.com/rust-lang/rust/issues/49359)).
16+
Currently, the overall automated testing experience spans multiple components owned by different
17+
teams across the Rust Project (e.g. T-cargo with `cargo test` being the primary
18+
touch point for most users, T-libs for libtest, T-rustdoc for doctests, etc.).
19+
This makes it more difficult to establish a vision and coordinate efforts to achieve that vision.
20+
However, there isn't any single team focused on the testing picture holistically.
21+
Simultaneously, there are a number of well known needs and feature requests in the space that do not have an explicit
22+
owner driving the efforts to completion.
1523

16-
Before this, there was a growing interest in improving testing generally, like [better CI integration](https://internals.rust-lang.org/t/pre-rfc-implementing-test-binary-list-format-json-for-use-by-ide-test-explorers-runners/18308).
24+
For example, there's been some long standing requests to have additional test output
25+
formats, such as JSON ([#49359]) and JUnit, available on Stable Rust.
26+
While some of these are available as unstable features in Nightly Rust,
27+
in [Rust 1.70.0](https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html), a bug was fixed
28+
so unstable test features, like `cargo test -- --format json`, require using a nightly
29+
toolchain, like other unstable features. This caused breakage in certain editor/IDE and CI
30+
related tooling as they had been relying on the prior behavior to get test data
31+
programmatically, and little progress has been made in the last 5 years ([#49359]).
1732

18-
One challenge with improving the situation is that a lot of concerns cross multiple teams, in particular cargo (for `cargo test`), libs (for libtest). This makes it more difficult to establish a vision and coordinate efforts to achieve that vision.
33+
Furthermore, there's been a growing interest in improving testing generally, like [better CI integration][ci] as well as requests for things like better support for custom test harnesses and frameworks (.e.g [#2318]).
1934

20-
There is also precedence for this today with t-devtools having more specific sub-teams like rustdoc and rustfmt.
35+
The new Testing team is intended to establish an overarching vision and provide focused attention on these areas.
36+
37+
[#49359]: https://github.com/rust-lang/rust/issues/49359
38+
[#50297]: https://github.com/rust-lang/rust/issues/50297
39+
[#2318]: https://github.com/rust-lang/rfcs/pull/2318
40+
[ci]: (https://internals.rust-lang.org/t/pre-rfc-implementing-test-binary-list-format-json-for-use-by-ide-test-explorers-runners/18308)
2141

2242
## Mission and responsibilities
2343
[mission]: #mission
2444

25-
This team would primarily be focused on iterating on the test writing experience, `cargo test`, and enabling integration with external tools like CI or IDEs.
45+
This team would be primarily focused on iterating on the test writing and reporting experience, `cargo test`, and enabling integration points and features for external tools like CI or IDEs.
2646

27-
Examples issues to resolve:
47+
Examples of issues to resolve:
2848
- [Stabilize support for programmatic (json) test output](https://github.com/rust-lang/rust/issues/49359)
2949
- What parts of [cargo nextest](https://nexte.st/) can we stabilize?
3050
- With the proliferation of test frameworks (e.g. [rstest], [trybuild], [trycmd], [cargo_test_support], [criterion]), are there underlying needs we can resolve?
3151

3252
## Relationships to other teams
3353

34-
T-devtools: This will be the parent team.
54+
With the aforementioned breadth across the Project, the Testing team will need to have collaborative relationships with many other teams, and is conceptually a subteam of both the Libs and Dev Tools teams.
55+
56+
The rust-lang/team repo does not currently support representing dual-parent subteams, so for now the Test team will be primarily under the Dev Tools team
57+
58+
T-devtools: This will be the primary top level parent team.
3559

3660
**T-cargo**: This is a sibling team that T-testing will need to work with similarly to T-rustfmt, T-clippy, etc.
3761

3862
**T-rustdoc**: This is a sibling team that T-testing will likely coordinate with if any changes are need to how we do doctesting
3963

40-
**T-libs**: This will effectively be another parent team as they are ultimately responsible for libtest.
64+
**T-libs**: This will be a second/secondary top level parent team as they are ultimately responsible for libtest.
4165

4266
## Processes
4367

4468
For decisions on vision and direction, T-testing will use a standard FCP process. T-testing will be subject to [T-cargo's processes](https://doc.crates.io/contrib/team.html#decision-process) when dealing with `cargo test` and T-libs's processes for libtest. For any newly developed crates and libraries, we will follow [T-cargo's processes](https://doc.crates.io/contrib/team.html#decision-process).
4569

4670
## Membership
4771

48-
Team members are expected to shepherd testing discussions and vote on FCPs and is independent of regular code contributions though contributing can help build up the relevant experience and/or demonstrate the qualities for team membership.
72+
Team members are expected to shepherd testing discussions and vote on FCPs. Team membership is independent of regular code contributions though contributing can help build up the relevant experience and/or demonstrate the qualities for team membership.
4973

5074
Qualifications that will be taken into account for membership include:
5175

5276
- Does the person have the judgement for when deciding when a larger consensus is needed?
5377
- Does the person understand the constraints of backwards compatibility within `cargo test` and libtest and exercise caution when extending the compatibly constraints?
5478

55-
Someone can become a member of the Cargo Team by requesting a review or being nominated by one of the existing members. They can be added by unanimous consent of the team. The team lead or another member of the team will also confirm with the moderation team that there are no concerns involving the proposed team member.
79+
Someone can become a member of the Testing Team by requesting a review or being nominated by one of the existing members. They can be added by unanimous consent of the team. The team lead or another member of the team will also confirm with the moderation team that there are no concerns involving the proposed team member.
5680

57-
Leads are responsible for scheduling and facilitating team meetings and will be selected from the team members by consensus.
81+
Team Leads are responsible for scheduling and facilitating team meetings and will be selected from the team members by consensus.
5882

5983
The initial members of the testing team shall be:
6084
- Lead: Caleb Cartwright (@calebcartwright)
6185
- Ed Page (@epage)
6286
- Weihang Lo (@weihanglo)
6387
- Scott Schafer (@Muscraft)
6488

65-
## Drawbacks and Alternatives
89+
## Drawbacks
90+
91+
The proposed Testing team bears some similarity to other Rust teams (e.g. Types team) in the sense
92+
that it would complicate and muddle the ownership of specific problems.
93+
For example, there would be two teams dealing with `cargo test` and two dealing with libtest.
94+
95+
## Rationale and alternatives
96+
97+
- This could be a working group instead of a team. However, we believe the [reasoning articulated in the Types team RFC][team-not-wg] is applicable here as well. There is a need for focused effort on driving work to completion along with associated maintenance work; not a shorter-lived initiative to create recommendations.
98+
- The Testing team could be a dual-parent subteam, but with the primary team under the Libs team. However, we believe Dev Tools is the better primary parent given the purview of the Testing team would extend well beyond libtest
99+
- The Testing team could be a single-parent subteam. We think there's too much overlap with too many teams across multiple top level teams to be a single-parent subteam.
100+
- We could do nothing and not form a new subteam nor a new working group. This would perpetuate the status quo and would most likely result in continued stagnation/lack of progress on the aforementioned focus areas.
66101

67-
- The ownership lines are becoming more muddled with the two teams dealing with `cargo test` and two teams dealing with libtest.
68102

103+
[team-not-wg]: https://rust-lang.github.io/rfcs/3254-types-team.html#why-a-team-and-not-a-working-group-what-is-the-difference-between-those-anyway
69104
[rstest]: https://crates.io/crates/rstest
70105
[trybuild]: https://crates.io/crates/trybuild
71106
[trycmd]: https://crates.io/crates/trycmd

0 commit comments

Comments
 (0)