Feature request: expand pylance to support IPython syntax (magics, etc).

[jakebailey]

Discussed in #1868

^{Originally posted by ryan-feeley September 23, 2021}
An IPython magic statement like %load_ext autoreload or %timeit f() is not valid pure python code, but it is valid IPython syntax, which one can use in the IPython shell or Jupyter notebook. These are popular development tools.

I believe would be useful enhancement for pylance to avoid giving errors on such IPython line and cell magics if the underlying file type in the editor is .ipy (standard extension for an IPython script file) or *.ipynb (jupyter notebook).

What is the current recommended workaround for such cases? Does one add a trailing # type: ignore to each line? Or does one break out of the zen of ipython and do something like

from IPython.core.getipython import get_ipython
get_ipython().run_line_magic('reload_ext', 'autoreload')

Is there any plan on the roadmap for pylance to be extended to IPython syntax?

[jakebailey]

@ryan-feeley Unless I'm mistaken, the Python/Jupyter extension should already be ignoring diagnostics for lines with magic on them. Is that not the case?

[ryan-feeley]

@jakebailey

Unless I'm mistaken, the Python/Jupyter extension should already be ignoring diagnostics for lines with magic on them. Is that not the case?

That is correct in jupyter notebook files, but not for *.ipy scripts open in the editor. See this image:

[jakebailey]

I haven't heard of .ipy files, only ipynb; how are you getting them to be treated as Python? Overriding VS Code's file associations?

There's special handling for notebooks for these IPython specific things, but these ipy files appear to be more like plopping notebook cells into plain Python files, except with a different grammar. Going so far as to parse another grammar without the cell ranges is going to be a bit tougher to deal with than us handling notebooks directly; this is much more like handling a new language that's derived from Python than anything else.

The Python/Jupyter extension IIRC support an "interactive mode", where these special directives are in comments instead, as an alternative for the time being.

[ryan-feeley]

VS Code automatically associates the Python language mode with .ipy files. It is a standard format, and using the IPython REPL in place of the in-box Python REPL is very common for developers, one reason being the support of the language extension magics. It is more or less the "recommended" python shell.

There are two flavors of IPython, one is the two process IPython kernel / frontend that is used by the .ipynb files you reference. The other is terminal-based IPython, which is the interactive interpreter you get by typing ipython at a prompt.

To do something like SHIFT+ENTER in the VS Code editor to run this code in the ipython shell, rather than the python shell in the terminal, you can just manually open the ipython shell instead, or you can do something like

    "python.terminal.launchArgs": [
        "-m",
        "IPython",
        "--no-autoindent",
    ],

in your settings.json.

I suppose you could also hack it and do something like less flexible like

"python.defaultInterpreterPath": "C:\\Users\\{$env:USERNAME}\\Anaconda3\\Scripts\\ipython.exe",

[ryan-feeley]

I get that this support might not be a near-term priority on your roadmap. It's just an FYI that the Pylance language server, which is also sort of playing the role of a linter, is flagging code that other linters which are aware of .ipy script files may not.

[jakebailey]

Pylance is a static analyzer for code only, and doesn't manage running code or anything, so I'm not too worried about that side.

I'm a bit surprised that VS Code would do this, but indeed it does:

https://github.com/microsoft/vscode/blob/main/extensions/python/package.json#L27
microsoft/vscode@c10ea17
microsoft/vscode#69694

All of the other files listed share the same Python grammar; ipy is the only one with an extended grammar. It's unusual that they would be treating this as more Python when it technically isn't. I wouldn't be surprised if this causes problems for other extensions too.

[ryan-feeley]

Thanks for digging into it. Yeah, it is a bit weird... But it's the nature of how some of these tools evolved over the years. It's mostly an issue in data science. If you doing actual software engineering, you wouldn't be just using grammar extensions willy-nilly. But if you're coming over from like PyCharm or Spyder which have always supported IPython magics, or from matlab, which has it's own magic, you might expect it to work.

I'm not sure you need to parse the grammar. Maybe just ignore it? By its very nature, these magics are sort of slip-streaming stuff in to the background that you probably don't need to be aware of. The "cell magic" syntax %%, which spans multiple lines might be harder to deal with. But the one-liners could perhaps just be white-listed.

[heejaechang]

@judej we need to discuss and decide how to support this in python sync meeting.

[heejaechang]

https://github.com/microsoft/pyrx/pull/2146 adds support for ipython magic in ipynb file. no support for ipy yet. any line with % in ipynb will be treated as non python code and ignored for now. no, intellisense support on magic line yet.

[timoklimmer]

@heejaechang @judej Is this for line magics only or does it also ignore cell magics? A cell magic is similar to a line magic except that it starts with %% and has multiple lines.

For example:

%%mylanguage

# just some arbitrary code in my new language, should be treated as non-python code

DROP ALL
CREATE NEW OBJECT NAMED xyz

It would be great to have cell magics support, too, as I am currently developing a cells magic, and all the hints from the Python linter are really disturbing.

[delgreen]

@heejaechang @judej Is this for line magics only or does it also ignore cell magics? A cell magic is similar to a line magic except that it starts with %% and has multiple lines.

For example:

%%mylanguage

# just some arbitrary code in my new language, should be treated as non-python code

DROP ALL
CREATE NEW OBJECT NAMED xyz

It would be great to have cell magics support, too, as I am currently developing a cells magic, and all the hints from the Python linter are really disturbing.

An Example of this occurring with ipython-sql

[timoklimmer]

@heejaechang @judej Any insights or comments?

[judej]

@timoklimmer, sorry no update as yet. We are working on updating our notebooks implementation and will be able to look at magics soon after that.

Moving this issue to discussion as an enhancement request for comments and upvotes.

[cspipaon]

@heejaechang @judej Are there any updates on the notebook implementation of Pylance? I'm specially looking forward to being able to rename symbols and use refactoring in notebooks.

This was being tracked in microsoft/vscode-jupyter/issues/5123 but I was redirected here. Please let me know if this is being tracked somewhere else.

[timoklimmer]

@judej @cspipaon I have set python.pylanceLspNotebooksEnabled to true, but I still get

Is this expected behavior or a bug?

[debonte]

This is currently expected. Ignoring everything in a cell after a %% cell magic sounds like good first step. @timoklimmer, can you open an issue on that? We'll keep this discussion open to track IPython syntax support, etc.

[ian-h-chamberlain]

Ironically, I came here looking to see if I could keep Pylance features on a cell with magic, for example if I have a cell like this:

%%capture

def xyz() -> str:
    print("hi")
    return 123

xyz()

The presence of %%capture seems to disable Pylance features (type checking, refactor, semantic highlight etc.) but in this case it would actually be preferable to treat everything but the magic as plain Python.

I'm not sure of a perfect way to solve this generally given the %%sql etc. use cases above, but there are two options that I think could work:

1. Support #!%% magic in notebook cells, which was the approach taken in IPython magic commands need to be runnable from comments so other tools can work with them. vscode-jupyter#3263 but it doesn't seem to work in .ipynb files (maybe it's just for .ipy files?). This would allow Pylance, isort etc. to just do their jobs easily without any special handling, I think.
2. Support %%capture # type:ignore or something along those lines, to indicate that the rest of the file should be treated as Python but ignore the %%. This seems perhaps more compatible with the default Jupyter implementation, but more work on the Pylance side (and other tools that might need to handle it)?

Should I open a new issue for this (and if so, on which repo)?

[debonte]

Thanks @ian-h-chamberlain. Yes, please open an issue in the Pylance repo. https://github.com/microsoft/pylance-release/issues/new/choose

[ian-h-chamberlain]

FYI, it's now been converted into a separate discussion #3851

[ShihengCao]

Is this same problem with this discussion?

this code can run correctly, but there is a prompt by pylance