Freeze when checking project

[qarmin]

Summary

Since about a week ago, whenever I open my project, I experience a very long (possibly indefinite) freeze with the following message:

The hotspot shows that some seemingly random functions are being executed continuously — mostly related to salsa and ty_python_semantic:

I suspect this might be related to the project's virtual environment (venv).

GDB shows a deep call stack with 1300 nested function calls - gdb.txt

Unfortunately, the project is internal, so I’m not able to share it and I don't know how debug this more

Version

ruff 161446a (happens since ~1 week)

[carljm]

Thanks for the report! If you have any more time to track this down, it would be useful to a) bisect the issue to a specific commit in ruff repo where it began, and b) narrow down a minimal reproducer by eliminating code from the project (or only checking specific files in the project) and see what makes the issue go away.

[qarmin]

Bisect points at - astral-sh/ruff#18041

[qarmin]

It turns out minimizing the code wasn’t as difficult as I expected

Steps to reproduce

• uv venv; uv pip install pandas
• Copy this to a.py

from pandas import DataFrame

class VFNorm(ABC):
    @property
    def df(self) -> DataFrame:
        return self._data

class SingleVisualFieldStatistics:
    def __init__(self, norm: VFNorm, age: float, values: list[PointValue]) -> None:
        self.model = norm.df.copy()

• ty check a.py

[AlexWaygood]

Simpler repro (requires uv >= 0.7.9):

1. Save this as foo.py in the ruff repo root:

   import pandas
   
   def f(x: pandas.DataFrame):
       y = x.copy()

2. Run uv run --no-project --with=pandas cargo run -p ty check foo.py from the Ruff repo root

[AlexWaygood]

It's the DataFrame._getitem_bool_array() method that's causing us to choke. This method is using an isinstance() check to narrow a parameter to an instance of pandas.core.Series and then accessing an attribute on the narrowed type. This is an even smaller repro:

from pandas.core.series import Series

def _getitem_bool_array(self, key):
    isinstance(key, Series) and key.index.equals(self)

(Needs further minimizing since Series is another huge pandas class, but that's all I'll have time for today I think.)

[AlexWaygood]

Smaller repro...

from pandas.core.indexes.base import Index

def _set_values(key):
    if isinstance(key, Index):
        key = key._values

[AlexWaygood]

I'm struggling to get much further than that.

[MichaReiser]

I'm struggling to get much further than that.

Are you saying, you're experiencing a hang too 😆

[sharkdp]

Running this with tracing shows that ty is not frozen/hanging. It "just" seems to be some kind of runtime problem. The logs show infer_expression_types calls inside pandas/core/indexes/base.py:

    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10a83) range=190240..190363 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10a86) range=190737..190769 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10955) range=154909..154965 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_definition_types with range=154901..154906 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10957) range=155073..155102 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(1095b) range=155028..155231 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10803) range=107725..107744 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(103fb) range=106751..106780 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(103ff) range=106669..106924 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10940) range=153254..153296 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10941) range=153387..153427 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_definition_types with range=153371..153376 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10942) range=153443..153460 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(10943) range=153443..153483 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(1093e) range=153058..153096 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/indexes/base.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(c1b8) range=159141..159241 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/series.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(c186) range=124885..124935 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/series.py"))
    in ty_python_semantic::types::infer::infer_definition_types with range=124868..124882 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/series.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(c188) range=125015..125075 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/series.py"))
    in ty_python_semantic::types::infer::infer_definition_types with range=125000..125012 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/series.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(c189) range=125088..125137 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/series.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(1af6) range=145701..145761 file=File(System("/home/shark/.cache/uv/archive-v0/qh6Cuh4I2HUnxuFLeH_PT/lib/python3.13/site-packages/pandas/core/frame.py"))
    in ty_python_semantic::types::infer::infer_expression_types with expression=Id(1800) range=52..60 file=File(System("/home/shark/issue_627/main.py"))
    in ty_python_semantic::types::infer::infer_definition_types with range=48..49 file=File(System("/home/shark/issue_627/main.py"))
    in ty_python_semantic::types::infer::infer_scope_types with scope=Id(1001) file=File(System("/home/shark/issue_627/main.py"))
    in ty_python_semantic::types::check_types with file=File(System("/home/shark/issue_627/main.py"))
    in ty_project::check_file with file=file(Id(c00))
    in ty_project::Project::check

Checking that file in isolation confirms that it is the source of the problem. Nothing immediately stands out, but this file contains one gigantic class Index. As I'm reducing the size of that class, the runtime slowly becomes better. Running samply on a somewhat minified version of that file shows that we're spending a lot of time looking up implicit instance attributes in that class, which seems to match well with the bisection result

[MichaReiser]

Does it repro both single and multi threaded? I guess the interesting question here is. Are we inferring the same attributes over and over again (some form of cache miss) or is it a very huge cycle or similar that takes forever to resolve

[sharkdp]

Does it repro both single and multi threaded?

Yes, this is just a single file. And it's not a hang, it actually finished if you give it enough time (or simplify the file).

I guess the interesting question here is. Are we inferring the same attributes over and over again (some form of cache miss) or is it a very huge cycle or similar that takes forever to resolve

I haven't investigated, but I'm assuming it's a problem where adding more methods and/or attributes to a class leads to a nonlinear increase in type check time, maybe because each attribute access requires going through all methods. And every method that we're adding is also an attribute, so maybe there's some quadratic growth? Just a wild guess.