Improve test-suite documentation (#7756)

- mypy_test and pyright no longer just test the stubs, they now also test other parts of typeshed as well.
- pytype_test.py can now be run on 3.10, meaning the whole test suite can now also be run on 3.10.
- Various test scripts now have from `__future__ import annotations`, meaning they can now only be run on 3.7+.
- Clean up the description of pyright_test.py, which had a slightly confusing wording.
- Also fix the `--dry-run` config option in mypy_test.py, which I accidentally broke in #7746

Author: AlexWaygood

.github/workflows/tests.yml

@@ -50,7 +50,7 @@ jobs:
       - run: ./tests/pytype_test.py --print-stderr
 
   mypy:
-    name: Run mypy against the stubs
+    name: Run mypy against typeshed
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -64,7 +64,7 @@ jobs:
       - run: ./tests/mypy_test.py --platform=${{ matrix.platform }} --python-version=${{ matrix.python-version }}
 
   pyright:
-    name: Run pyright against the stubs
+    name: Run pyright against typeshed
     runs-on: ubuntu-latest
     strategy:
       matrix:

tests/README.md

@@ -13,29 +13,31 @@ objects at runtime.
 objects at runtime.
 
 To run the tests, follow the [setup instructions](../CONTRIBUTING.md#preparing-the-environment)
-in the `CONTRIBUTING.md` document. In particular, we recommend running with Python 3.8 or 3.9.
+in the `CONTRIBUTING.md` document. In particular, we recommend running with Python 3.8+.
 
 ## mypy\_test.py
 
-This test requires Python 3.6 or higher; Python 3.6.1 or higher is recommended.
-Run using:
+This test requires Python 3.7+. Run using:
 ```
 (.venv3)$ python3 tests/mypy_test.py
 ```
 
-This test is shallow — it verifies that all stubs can be
+The test has four parts. Each part uses mypy with slightly different configuration options:
+- Running mypy on the stdlib stubs
+- Running mypy on the third-party stubs
+- Running mypy `--strict` on the scripts in the `tests` directory
+- Running mypy `--strict` on the regression tests in the `test_cases` directory.
+
+When running mypy on the stubs, this test is shallow — it verifies that all stubs can be
 imported but doesn't check whether stubs match their implementation
-(in the Python standard library or a third-party package). It has an exclude list of
-modules that are not tested at all, which also lives in the tests directory.
+(in the Python standard library or a third-party package).
 
-You can restrict mypy tests to a single version by passing `-p2` or `-p3.9`:
-```bash
-(.venv3)$ python3 tests/mypy_test.py -p3.9
-```
+Run `python tests/mypy_test.py --help` for information on the various configuration options
+for this test script.
 
 ## pytype\_test.py
 
-This test requires Python between 3.6 and 3.9.
+This test requires Python 3.7+.
 Note: this test cannot be run on Windows
 systems unless you are using Windows Subsystem for Linux.
 
@@ -48,18 +50,19 @@ This test works similarly to `mypy_test.py`, except it uses `pytype`.
 
 ## pyright\_test.py
 
-This test requires [Node.js](https://nodejs.org) to be installed. It is
-currently not part of the CI,
-but it uses the same pyright version and configuration as the CI.
+This test requires [Node.js](https://nodejs.org) to be installed. Although
+typeshed runs pyright in CI, it does not currently use this script. However,
+this script uses the same pyright version and configuration as the CI.
 ```
 (.venv3)$ python3 tests/pyright_test.py                                # Check all files
 (.venv3)$ python3 tests/pyright_test.py stdlib/sys.pyi                 # Check one file
 (.venv3)$ python3 tests/pyright_test.py -p pyrightconfig.stricter.json # Check with the stricter config.
 ```
 
 `pyrightconfig.stricter.json` is a stricter configuration that enables additional
-checks that would typically fail on incomplete stubs (such as `Unknown` checks),
-and is run on a subset of stubs (including the standard library).
+checks that would typically fail on incomplete stubs (such as `Unknown` checks).
+In typeshed's CI, pyright is run with these configuration settings on a subset of
+the stubs in typeshed (including the standard library).
 
 ## check\_consistent.py
 
@@ -102,7 +105,7 @@ stubtest can also help you find things missing from the stubs.
 
 ## stubtest\_third\_party.py
 
-This test requires Python 3.6 or higher.
+This test requires Python 3.7+.
 Run using
 ```
 (.venv3)$ python3 tests/stubtest_third_party.py

tests/mypy_test.py

@@ -358,7 +358,10 @@ def test_the_test_scripts(code: int, major: int, minor: int, args: argparse.Name
     flags = get_mypy_flags(args, major, minor, None, strict=True, test_suite_run=True)
     print(f"Testing the test suite ({num_test_files_to_test} files)...")
     print("Running mypy " + " ".join(flags))
-    this_code = subprocess.run([sys.executable, "-m", "mypy", "tests", *flags]).returncode
+    if args.dry_run:
+        this_code = 0
+    else:
+        this_code = subprocess.run([sys.executable, "-m", "mypy", "tests", *flags]).returncode
     code = max(code, this_code)
     return TestResults(code, num_test_files_to_test)
 
@@ -369,7 +372,10 @@ def test_the_test_cases(code: int, major: int, minor: int, args: argparse.Namesp
     flags = get_mypy_flags(args, major, minor, None, strict=True, custom_typeshed=True)
     print(f"Running mypy on the test_cases directory ({num_test_case_files} files)...")
     print("Running mypy " + " ".join(flags))
-    this_code = subprocess.run([sys.executable, "-m", "mypy", "test_cases", *flags]).returncode
+    if args.dry_run:
+        this_code = 0
+    else:
+        this_code = subprocess.run([sys.executable, "-m", "mypy", "test_cases", *flags]).returncode
     code = max(code, this_code)
     return TestResults(code, num_test_case_files)