[ty] Treat Callables as bound-method descriptors in special cases
#20802
+217
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Diagnostic diff on typing conformance testsNo changes detected when running ty on typing conformance tests ✅ |
|
sharkdp
changed the title
Callables as bound-method descripors in special casesCallables as bound-method descriptors in special cases
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
unsupported-operator |
0 | 2,961 | 0 |
missing-argument |
0 | 37 | 0 |
unresolved-attribute |
0 | 3 | 0 |
possibly-missing-attribute |
0 | 2 | 0 |
unused-ignore-comment |
2 | 0 | 0 |
invalid-assignment |
1 | 0 | 0 |
no-matching-overload |
0 | 1 | 0 |
| Total | 3 | 3,004 | 0 |
sharkdp
force-pushed
the
david/callables-as-descriptors
branch
from
f2e2b2f to
3dc91d1
Compare
sharkdp
commented
Oct 13, 2025
| python-version = "3.14" | ||
| ``` | ||
|
|
||
| ## Introduction |
There was a problem hiding this comment.
Part of the text here is a copy from @carljm's post here: https://discuss.python.org/t/when-should-we-assume-callable-types-are-method-descriptors/92938
sharkdp
commented
Oct 13, 2025
sharkdp
force-pushed
the
david/callables-as-descriptors
branch
from
3dc91d1 to
bf21fb5
Compare
sharkdp
commented
Oct 13, 2025
Comment on lines
+166
to
+190
| The reason for this is that the heuristic is problematic. We don't *know* that the `Callable` in the | ||
| return type of `memoize` is actually related to the method that we pass in. But when `memoize` is | ||
| applied as a decorator, it is reasonable to assume so. | ||
|
|
||
| In general, a function call might however return a `Callable` that is unrelated to the argument | ||
| passed in. And here, it seems more reasonable and safe to treat the `Callable` as a non-descriptor. | ||
| This allows correct programs like the following to pass type checking (that are currently rejected | ||
| by pyright and mypy with a heuristic that apparently applies in a wider range of situations): | ||
|
|
||
| ```py | ||
| class SquareCalculator: | ||
| def __init__(self, post_process: Callable[[float], int]): | ||
| self.post_process = post_process | ||
|
|
||
| def __call__(self, x: float) -> int: | ||
| return self.post_process(x * x) | ||
|
|
||
| def square_then(c: Callable[[float], int]) -> Callable[[float], int]: | ||
| return SquareCalculator(c) | ||
|
|
||
| class Calculator: | ||
| square_then_round = square_then(round) | ||
|
|
||
| reveal_type(Calculator().square_then_round(3.14)) # revealed: Unknown | int | ||
| ``` |
There was a problem hiding this comment.
The argument here is a bit weak. I had trouble coming up with a better example.
It still feels wrong to me to generally assume that returned Callable types are function-like if a function-like callable is passed into a call (would we try to detect that in every possible parameter position? or only apply the heuristic for single-argument calls?).
There was a problem hiding this comment.
The most common example where a non-function callable is returned by a decorator is functools._lru_cache_wrapper:
% py
Python 3.13.1 (main, Jan 3 2025, 12:04:03) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import functools
>>> class F:
... @functools.lru_cache
... def whatever(self): ...
...
>>> F.whatever
<functools._lru_cache_wrapper object at 0x101549640>But (somewhat famously), _lru_cache_wrapper has the same descriptor-binding behaviour that types.FunctionType has. So you could use that as an argument in favour of assuming that all objects returned from decorators have this method-binding behaviour if they have __call__ attributes, I suppose...
sharkdp
commented
Oct 13, 2025
sharkdp
requested review from
AlexWaygood,
carljm and
dcreager
as code owners
AlexWaygood
reviewed
Oct 13, 2025
AlexWaygood
reviewed
Oct 13, 2025
AlexWaygood
reviewed
Oct 13, 2025
AlexWaygood
reviewed
Oct 13, 2025
AlexWaygood
reviewed
Oct 13, 2025
There was a problem hiding this comment.
Nearly all the ecosystem diff appears to be due to the bind_self fix in signatures.rs, which actually seems unrelated to the rest of the PR... I wonder if we could separate that out into a standalone PR, so that we can see the impact of the special case here more clearly?
crates/ty_python_semantic/resources/mdtest/call/callables_as_descriptors.md
Outdated
Show resolved
Hide resolved
Comment on lines
+23
to
+28
| def _( | ||
| accessed_on_class: CallableTypeOf[C1.method], | ||
| accessed_on_instance: CallableTypeOf[C1().method], | ||
| ): | ||
| reveal_type(accessed_on_class) # revealed: (self: C1, x: int) -> str | ||
| reveal_type(accessed_on_instance) # revealed: (x: int) -> str |
There was a problem hiding this comment.
off-topic: a reveal_typeform function would be pretty useful for this kind of mdtest (and for playing around in the playground). Rather than having to create the function, I could just do reveal_typeform(CallableTypeOf[C1.method]) and it would print the signature in a diagnostic
There was a problem hiding this comment.
a reveal_typeform function would be pretty useful
Yes!!
There was a problem hiding this comment.
Or if Python had explicitly specialized calls to generic functions, we could finally add C++'s std::declval:
def declval[T]() -> T:
raise RuntimeError("can not make up a `T` out of thin air")
reveal_type(declval[CallableTypeOf[C1.method]]())Beautiful!
Comment on lines
+31
to
+33
| Other callable objects (`staticmethod` objects, instances of classes with a `__call__` method but no | ||
| dedicated `__get__` method) are *not* bound-method descriptors. If accessed as class attributes via | ||
| an instance, they are simply themselves: |
There was a problem hiding this comment.
staticmethod does actually have a __get__ method at runtime -- it returns the underlying function object (whether it's accessed on the class or the instance). On Python <=3.9, staticmethods are not directly callable; it's only by calling __get__ on the staticmethod that you retrieve the original function object and get something that you can actually call:
% uvx python3.9
Python 3.9.6 (default, Apr 30 2025, 02:07:17)
[Clang 17.0.0 (clang-1700.0.13.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
... @staticmethod
... def bar(): return 42
...
>>> original_staticmethod = Foo.__dict__['bar']
>>> original_staticmethod
<staticmethod object at 0x100575370>
>>> original_staticmethod()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'staticmethod' object is not callable
>>> Foo.bar
<function Foo.bar at 0x1005fb4c0>
>>> Foo.bar()
42On Python 3.10, staticmethod.__call__ was added, so staticmethod objects are now directly callable. But it's still the case that staticmethods are descriptors that return the underlying function object when you access them, on either the class object or an instance of the class.
There was a problem hiding this comment.
staticmethoddoes actually have a__get__method at runtime -- it returns the underlying function object
Right, I know. The text says "Other callable objects (staticmethod objects, […]) are not bound-method descriptors", where "bound-method descriptor" would imply the sort of self-binding behavior that we know from normal functions. But the text could certainly be clarified further, if you think that'd be valuable.
On Python 3.10,
staticmethod.__call__was added, sostaticmethodobjects are now directly callable.
TIL!
There was a problem hiding this comment.
The text says "Other callable objects (
staticmethodobjects, […]) are not bound-method descriptors"
Hmmmm, but it also says:
If accessed as class attributes via
an instance, they are simply themselves
Which I don't think is accurate for staticmethods — they are descriptors, just not in the same way as instance methods
Comment on lines
+166
to
+190
| The reason for this is that the heuristic is problematic. We don't *know* that the `Callable` in the | ||
| return type of `memoize` is actually related to the method that we pass in. But when `memoize` is | ||
| applied as a decorator, it is reasonable to assume so. | ||
|
|
||
| In general, a function call might however return a `Callable` that is unrelated to the argument | ||
| passed in. And here, it seems more reasonable and safe to treat the `Callable` as a non-descriptor. | ||
| This allows correct programs like the following to pass type checking (that are currently rejected | ||
| by pyright and mypy with a heuristic that apparently applies in a wider range of situations): | ||
|
|
||
| ```py | ||
| class SquareCalculator: | ||
| def __init__(self, post_process: Callable[[float], int]): | ||
| self.post_process = post_process | ||
|
|
||
| def __call__(self, x: float) -> int: | ||
| return self.post_process(x * x) | ||
|
|
||
| def square_then(c: Callable[[float], int]) -> Callable[[float], int]: | ||
| return SquareCalculator(c) | ||
|
|
||
| class Calculator: | ||
| square_then_round = square_then(round) | ||
|
|
||
| reveal_type(Calculator().square_then_round(3.14)) # revealed: Unknown | int | ||
| ``` |
There was a problem hiding this comment.
The most common example where a non-function callable is returned by a decorator is functools._lru_cache_wrapper:
% py
Python 3.13.1 (main, Jan 3 2025, 12:04:03) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import functools
>>> class F:
... @functools.lru_cache
... def whatever(self): ...
...
>>> F.whatever
<functools._lru_cache_wrapper object at 0x101549640>But (somewhat famously), _lru_cache_wrapper has the same descriptor-binding behaviour that types.FunctionType has. So you could use that as an argument in favour of assuming that all objects returned from decorators have this method-binding behaviour if they have __call__ attributes, I suppose...
AlexWaygood
approved these changes
Oct 13, 2025
sharkdp
force-pushed
the
david/callables-as-descriptors
branch
from
7110b6b to
cd04d4b
Compare
AlexWaygood
reviewed
Oct 13, 2025
| Ok(return_ty) => { | ||
| let is_input_function_like = inferred_ty | ||
| .into_callable(self.db()) | ||
| .and_then(Type::unwrap_as_callable_type) |
There was a problem hiding this comment.
Note that unwrap_as_callable_type will return None if into_callable returns Some(Type::Union), even if each element of the union returned was a Callable type
There was a problem hiding this comment.
Oh, that's a good catch! Will look into this tomorrow (post-merge, I don't think it's critical).
dcreager
added a commit
that referenced
this pull request
Oct 13, 2025
…tity * origin/main: (24 commits) Update Python compatibility from 3.13 to 3.14 in README.md (#20852) [syntax-errors]: break outside loop F701 (#20556) [ty] Treat `Callable`s as bound-method descriptors in special cases (#20802) [ty] Do not bind self to non-positional parameters (#20850) Fix syntax error false positives on parenthesized context managers (#20846) [ty] Remove 'pre-release software' warning (#20817) Render unsupported syntax errors in formatter tests (#20777) [ty] Treat functions, methods, and dynamic types as function-like `Callable`s (#20842) [ty] Move logic for `super()` inference to a new `types::bound_super` submodule (#20840) [ty] Fix false-positive diagnostics on `super()` calls (#20814) [ty] Move `class_member` to `member` module (#20837) [`ruff`] Use DiagnosticTag for more flake8 and numpy rules (#20758) [ty] Prefer declared base class attribute over inferred attribute on subclass (#20764) [ty] Log files that are slow to type check (#20836) Update cargo-bins/cargo-binstall action to v1.15.7 (#20827) Update CodSpeedHQ/action action to v4.1.1 (#20828) Update Rust crate pyproject-toml to v0.13.7 (#20835) Update Rust crate anstream to v0.6.21 (#20829) Update Rust crate libc to v0.2.177 (#20832) Update Rust crate memchr to v2.7.6 (#20834) ...
dcreager
added a commit
that referenced
this pull request
Oct 13, 2025
* main: (25 commits) [ty] Diagnostic for generic classes that reference typevars in enclosing scope (#20822) Update Python compatibility from 3.13 to 3.14 in README.md (#20852) [syntax-errors]: break outside loop F701 (#20556) [ty] Treat `Callable`s as bound-method descriptors in special cases (#20802) [ty] Do not bind self to non-positional parameters (#20850) Fix syntax error false positives on parenthesized context managers (#20846) [ty] Remove 'pre-release software' warning (#20817) Render unsupported syntax errors in formatter tests (#20777) [ty] Treat functions, methods, and dynamic types as function-like `Callable`s (#20842) [ty] Move logic for `super()` inference to a new `types::bound_super` submodule (#20840) [ty] Fix false-positive diagnostics on `super()` calls (#20814) [ty] Move `class_member` to `member` module (#20837) [`ruff`] Use DiagnosticTag for more flake8 and numpy rules (#20758) [ty] Prefer declared base class attribute over inferred attribute on subclass (#20764) [ty] Log files that are slow to type check (#20836) Update cargo-bins/cargo-binstall action to v1.15.7 (#20827) Update CodSpeedHQ/action action to v4.1.1 (#20828) Update Rust crate pyproject-toml to v0.13.7 (#20835) Update Rust crate anstream to v0.6.21 (#20829) Update Rust crate libc to v0.2.177 (#20832) ...
dcreager
added a commit
that referenced
this pull request
Oct 14, 2025
…rable * origin/main: (26 commits) [ty] Add separate type for typevar "identity" (#20813) [ty] Diagnostic for generic classes that reference typevars in enclosing scope (#20822) Update Python compatibility from 3.13 to 3.14 in README.md (#20852) [syntax-errors]: break outside loop F701 (#20556) [ty] Treat `Callable`s as bound-method descriptors in special cases (#20802) [ty] Do not bind self to non-positional parameters (#20850) Fix syntax error false positives on parenthesized context managers (#20846) [ty] Remove 'pre-release software' warning (#20817) Render unsupported syntax errors in formatter tests (#20777) [ty] Treat functions, methods, and dynamic types as function-like `Callable`s (#20842) [ty] Move logic for `super()` inference to a new `types::bound_super` submodule (#20840) [ty] Fix false-positive diagnostics on `super()` calls (#20814) [ty] Move `class_member` to `member` module (#20837) [`ruff`] Use DiagnosticTag for more flake8 and numpy rules (#20758) [ty] Prefer declared base class attribute over inferred attribute on subclass (#20764) [ty] Log files that are slow to type check (#20836) Update cargo-bins/cargo-binstall action to v1.15.7 (#20827) Update CodSpeedHQ/action action to v4.1.1 (#20828) Update Rust crate pyproject-toml to v0.13.7 (#20835) Update Rust crate anstream to v0.6.21 (#20829) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Treat
Callables as bound-method descriptors ifCallableis the return type of a decorator that is applied to a function definition. See the rendered version of the new test file for the full description of this new heuristic.I could imagine that we want to treat
Callables as bound-method descriptors in other cases as well, but this seems like a step in the right direction. I am planning to add other "use cases" from astral-sh/ty#491 to this test suite.partially addresses astral-sh/ty#491
closes astral-sh/ty#1333
Ecosystem impact
All positive
unsupported-operatordiagnostics onsympy, which was one of the main motivations for implementing this changemissing-argumentdiagnostics, and no added call-error diagnostics, which is an indicator that this heuristic shouldn't cause many false positivespossibly-missing-attributediagnostics when accessing attributes like__name__on decorated functions. The two addedunused-ignore-commentdiagnostics are also cases of this.invalid-assignmentdiagnostic ondd-trace-py, which looks suspicious, but only because ourinvalid-assignmentdiagnostics are not great. This is actually a "Implicit shadowing of function" diagnostic that hides behind theinvalid-assignmentdiagnostic, because a module-global function is being patched through amodule.funcattribute assignment.Test Plan
New Markdown tests.