Make the file count limit for indexing configurable

[LeBoucEtMistere]

Environment data

• Language Server version: 2022.6.11, vscode over ssh
• OS and version: Ubuntu 20.04.3 LTS
• Python version (& distribution if applicable, e.g. Anaconda): Python 2.7.18 (Proprietary codebase, cannot give more details on packages and such)

Pylance config:

{
    "python.analysis.indexing": true,
    "python.languageServer": "Pylance",
    "python.formatting.provider": "black",
    "python.analysis.useLibraryCodeForTypes": false
}

Problem statement

Running Pylance through the official extension, in a large proprietary Python Monorepo, in a python2 virtualenv. Symbol search (Cmd+T) takes more than a minute to find symbols. However Cmd+Clicking a symbol in the code is working instantaneously. From the logs I can read, the index size reaches its limit causing VSCode to drop index cache.

Repro Steps

Repo I use is private and proprietary, cannot share any reproduction steps. At least 13k python files in the repo.

Expected behavior

Symbol search being fast. In particular, PyCharm is able to handle this repo properly and its symbol search is working as expected while VSCode is not.

Actual behavior

Symbol search unusably slow

Logs

[Info  - 8:35:38 AM] (708066) Pylance language server 2022.6.11 (pyright bbc42826) starting
[Info  - 8:35:38 AM] (708066) Server root directory: /home/arthur.depasse/.vscode-server/extensions/ms-python.vscode-pylance-2022.6.11/dist
Notebook support: Legacy
[Info  - 8:35:38 AM] (708066) No configuration file found.
[Info  - 8:35:38 AM] (708066) pyproject.toml file found at <redacted-repo>.
[Info  - 8:35:38 AM] (708066) Setting pythonPath for service "<redacted-repo>": "/home/arthur.depasse/<obfuscated-repo>/venv2/bin/python"
[Info  - 8:35:38 AM] (708066) Loading pyproject.toml file at /home/arthur.depasse/<redacted-repo>/pyproject.toml
[Error - 8:35:38 AM] (708066) Pyproject file "/home/arthur.depasse/<redacted-repo>/pyproject.toml" is missing "[tool.pyright]" section.
[Warn  - 8:35:38 AM] (708066) stubPath /home/arthur.depasse/<redacted-repo>/typings is not a valid directory.
[Info  - 8:35:38 AM] (708066) Python version 2.7 from interpreter is unsupported
[Info  - 8:35:38 AM] (708066) Assuming Python platform Linux
[Info  - 8:35:39 AM] (708066) Searching for source files
[Info  - 8:35:39 AM] (708066) Auto-excluding /home/arthur.depasse/<redacted-repo>/venv2
[Info  - 8:35:40 AM] (708066) Found 12970 source files
[Info  - 8:35:40 AM] (708066) Background analysis(1) root directory: /home/arthur.depasse/.vscode-server/extensions/ms-python.vscode-pylance-2022.6.11/dist
[Info  - 8:35:40 AM] (708066) Background analysis(1) started
[Info  - 8:35:40 AM] (708066) Indexer background runner(2) root directory: /home/arthur.depasse/.vscode-server/extensions/ms-python.vscode-pylance-2022.6.11/dist (index)
[Info  - 8:35:40 AM] (708066) Indexing(2) started
[Info  - 8:35:44 AM] (708066) [BG(1)] Long operation: getSemanticTokens full at /home/arthur.depasse/<redacted-repo>/<redacted-file>.py (2773ms)
[Info  - 8:35:44 AM] (708066) [IDX(2)] Long operation: index execution environment /home/arthur.depasse/<redacted-repo> (3302ms)
[Info  - 8:35:44 AM] (708066) [IDX(2)] Long operation: index libraries /home/arthur.depasse/<redacted-repo> (4293ms)
[Info  - 8:35:44 AM] (708066) Indexer done(2). indexed 516 files
[Warn  - 8:36:55 AM] (708066) Workspace indexing has hit its upper limit: 2000 files

[bschnurr]

could try disabling indexing or using a pyrightconfig.json file to exclude folders
https://github.com/microsoft/pyright/blob/main/docs/configuration.md#sample-config-file

[LeBoucEtMistere]

So I added an include and exclude section to my pyproject file, excluded as much stuff as I could, and went from 13k files to 8.7k files analyzed by Pylance. It's still well above the 2k limit for indexing. This resulted in a small speedup in symbol search but not going below the 30sec bar, and far from being usable. I also tried disabling indexing and it didn't really helped.

Since working in repos that contain more than 2k files that need to be analyzed and indexed is pretty much a reality for anyone working in a big monorepo, is there a way to raise this 2k files index limit ? I wouldn't mind paying the cost of indexing once when I start the server and then have it cached on file or in memory even if the cache gets really big, as long as it makes search features usable

[LeBoucEtMistere]

In fact that is the current behaviour of Pycharm and probably one of the last things that prevent me from moving away from it to VSC. It indexes your whole repo in a single pass when the language server starts and then saves a complete cache of the index. You also get visual feedback on the indexing operation (which is not the case with Pylance and makes it hard to understand what is happening when)

[judej]

Thanks for the feedback, unfortunately, lazy analysis is fundamental to the design of Pyright - the analysis engine that Pylance uses. It deliberately does not cache the analysis results. I don't see this changing in Pyright anytime soon.

[ZFYL]

I think it's only performance related. Maybe we could look under the hood of VSCode and force overwrite the 2000 limit? Any developments on your side?

[ZFYL]

i found the relevant code snippet

it's hard coded into VSCode we cannot change it easily

[ZFYL]

this is the path

[shivk012]

If you want to try it locally (and will have to redo it each time the extension updates you can manually override it in the pyright bundle)
e.g. ~/.vscode/extensions/ms-python.vscode-pylance-2023.2.21/dist
search for 2e3 and replace it with something you want instead

[santagada]

oh, that's easy, lets make an extension that patches this extension

[santagada]

It says this issue is waiting for response from the user. What response is needed?

[LeBoucEtMistere]

Yeah I think the original question is still waiting for a clear answer here, will you add support for indexing more than 2k files? I don't mind pyright using lazy analysis as long as I can force it to keep more files analyzed in memory

[erictraut]

@judej & @heejaechang, what concerns do you have (if any) about exposing a configuration option that controls the file count limit for indexing? The 2K default was chosen to limit indexing time to under a minute in most cases. Obviously, indexing more files will take more time when the project is loaded, but if the user is willing to incur this additional cost and time (say, 2 or 3 minutes), that should be their prerogative, no? I don't see memory being a limiting factor here since the in-memory index takes up relatively little space.

[santagada]

Done, a simple fix for people that don't care about the reasons for the limit, save this to a script and run it... it should fix latest and all other versions of the extension

import re
import sys
from pathlib import Path

pyright_relative_path = Path('dist/pyright.bundle.js')
vscode_extension_path = Path('~/.vscode/extensions').expanduser()

if not vscode_extension_path.exists():
    print(f'Cannot find vscode extension path, its not in {vscode_extension_path}')
    sys.exit(1)
    
pylance_extensions = list(vscode_extension_path.glob('ms-python.vscode-pylance-*'))

if len(pylance_extensions) == 0:
    print(f'Cannot find any pylance extensions inside vscode extension path {vscode_extension_path}')
    sys.exit(1)

patched = False
for ext in pylance_extensions:
    pyright_file = ext / pyright_relative_path
    if not pyright_file.exists():
        print(f'Extension {ext} did not have a {pyright_relative_path} file, could not patch it')
        continue
    text = pyright_file.read_text()
    # this starts with a digit because there's another line setting it to viod 0;
    text = re.sub(r'\.MaxWorkspaceIndexFileCount=(\d[^;]+);', '.MaxWorkspaceIndexFileCount=2e6;', text)
    pyright_file.write_text(text)
    print(f'Replaced the max files in {pyright_file} to {2e6}')

print('All done!')

[heejaechang]

let me ask team on this. usually adding new option, we wait how many people want it. so far we didn't get much upvote so that's why there is no traction on this. but actual change is quite simple. so let me ask PM and team again on how they think.

[shivk012]

Thank you!

[ZFYL]

Thank you for both of you! This is a blocker for our 300.000+ LOC codebase 🥳

[zacker150]

@heejaechang, any update from the PM and team?

[rchiodo]

The indexing limit was added. The setting is called:

{
    "python.analysis.userFileIndexingLimit": 2000
}

It defaults to 2000.