mypy `install_types` option causes silent dependency conflicts

[jtherrmann]

Jira: https://asfdaac.atlassian.net/browse/TOOL-3482

Note: The above link is accessible only to members of ASF.

---

Description

Our recommended mypy options currently include:

install_types = true
non_interactive = true
disable_error_code = ["import-untyped"]

The install_types option tells mypy to automatically install missing type stub packages for third-party libraries using pip. Unfortunately, this can cause an error like the following:

Installing collected packages: urllib3, types-requests
  Attempting uninstall: urllib3
    Found existing installation: urllib3 1.26.19
    Uninstalling urllib3-1.26.19:
      Successfully uninstalled urllib3-1.26.19
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
botocore 1.35.94 requires urllib3<1.27,>=1.25.4; python_version < "3.10", but you have urllib3 2.3.0 which is incompatible.

Our current solution is to add the offending type stub packages (in this case, types-requests) to the project requirements so that they get installed when building the environment.

However, I recently discovered that if the above pip error occurs when running mypy, it does not cause a failure in either the mypy command or the GitHub Actions workflow. This means that we'll likely only notice this error when running mypy locally, so we could have unnoticed dependency conflicts in our GitHub Actions mypy workflow.

This is probably not a major problem, since it appears that mypy is still able to perform type checking despite the above error. But it's something we should fix by removing the install_types option (at which point we can also remove non_interactive, since it only affects install_types), and adding the stub packages to our project dependencies.

This raises another issue, which is that if you disable install_types, then mypy is supposed to warn you about third-party libraries that have stub packages available to install, e.g:

error: Library stubs not installed for "requests"  [import-untyped]
note: Hint: "python3 -m pip install types-requests"

However, as of mypy 1.14.1, the import-untyped error code is also used for untyped imports without available stub packages, e.g:

error: Skipping analyzing "boto3": module is installed, but missing library stubs or py.typed marker  [import-untyped]

This is why our recommended mypy config includes the disable_error_code = ["import-untyped"] option: because we want to ignore errors for untyped imports where there is no stub package available. But this has the unintended consequence of also ignoring errors for untyped imports where there is a stub package available. So, if we remove the install_types option, we should also remove the disable_error_code = ["import-untyped"] option, so that mypy will show us which third-party libraries have stub packages available. Then we'll also have to individually ignore any untyped imports without available stub packages.

Also see:

• Option "install-types" breaks project version dependencies. python/mypy#17852
• https://github.com/ASFHyP3/.github-private/wiki/Mypy#missing-imports
• https://github.com/ASFHyP3/.github-private/wiki/Mypy#failure-to-automatically-install-type-stubs
• https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
• https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-install-types
• https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-non-interactive

TODO

• Remove the install_types, non_interactive, and disable_error_code options shown above from our recommended mypy options and from our hyp3-cookiecutter mypy config
• Update each of our existing repos:
  • Remove those mypy options
  • Run mypy to show import-untyped errors
  • For untyped imports with available stub packages, add the stub packages to our project requirements (wherever mypy is listed)
  • For untyped imports without available stub packages, ignore those errors individually
• Update the related wiki article sections: Missing imports and Failure to automatically install type stubs

[jtherrmann]

I opened a mypy feature request related to this: python/mypy#18661