DAP client/server symmetry and command line interfaces

[lanza]

I worked on lldb and IDE integration centered around the DAP and lldb-vscode last year. One glaring problem that I found was the dap-repl (as this plugin has termed it) is very limited in functionality. e.g. it doesn't support vt220/ecma-48/man console_codes and is thus a (very) dumb terminal.

If you're a systems programmer (or any other programmer that prefers the terminal) this is a big issue since you probably learned to debug via command line interfaces. So as a systems engineer working on systems tooling for other systems engineer I wanted to properly introduce a mechanism to let the command gdb/lldb command prompt run the show and have the DAP operate more as a GUI library instead of debugger controller.

So, sorry for the long exposition, but I've had problems explaining this part to people.

What I'm aiming to do is to add extensions to the DAP to enable this to happen. As an exemplary usage flow:

(in vim) :StartDapLldbVscode

would take you to a split screen with the left side an empty buffer and the right side a neovim terminal emulator window hosting lldb's command interpreter. You type file a.out, b main, r and it starts as usual but the DAP is used more-or-less as a GUI slave that can forward user actions such as breakpoint/step/continue/evaluate requests.

The only real requirement here would be for the DAP client to acknowledge that it can support a PTY pair hosting terminal for the DAP server to attach to the PTY slave with some proper TERM var set and function as normal and then to actually offer the connection. Luckily for neovim's purposes, we can create arbitrary terminals for free and just spawn the DAP server there and inform it that this mode is supported.

For further reading, I tried submitted this idea to the DAP official repository but didn't get far. microsoft/debug-adapter-protocol#45

I mention this here because I actually just created my own project called nvim-dap about two weeks ago to tinker around with this before inadvertently seeing this mentioned on reddit. So I figured I'd check if the idea sounded interesting.

[theHamsta]

Reading the source code of lldb-vscode, it appears you can set to run it in terminal in the start request. The option only apeared recently (I had to git pull to find it (git blame says 2020-09-01). I didn't try it.

runInTerminal = True

There's highlighting schemes for vim for terminal codes.

[lanza]

That's running the inferior in the terminal, not the debug adapter. The actual code necessary for this to work would be in this diff on llvm reviews:

https://reviews.llvm.org/D61476

[mfussenegger]

I'm not sure I understand everything right, so let me just recap what I understood.

Given the following definitions:

+------------+       +----------------+         +-----------+       +------------+
| DAP Client |<----->| Debug Adapter  |<------->| Debugger  |<----->| Debugee    |
| (nvim-dap) |       | (DAP server)   |         |           |       | (your app) |
+------------+       +----------------+         +-----------+       +------------+

You'd like to spawn the debugger in a terminal inside neovim, so that you could use it's more powerful command prompt as a user interface?
runInTerminal doesn't work, because it is used to launch the debugee, not the debugger.

Would it be possible to instead use the attach mechanism of the protocol? Basically delegate the responsibility of launching the debug adapter and debugger. I'm not sure how lldb-vscode / lldb work, but if there were the option to launch lldb in a terminal and additionally have DAP over TCP, then in Neovim that could be launched via :terminal and users could connect to it using a adapter configuration of type server.

Then the remaining question would be what "GUI-slave" functionality would be expected. Given the breakpoint events the editor could at least indicate breakpoints and would stop to the breakpoints on stopped events.

update

Looks like this has already been suggested here microsoft/debug-adapter-protocol#45 (comment)

[lanza]

So my interest here isn't really in just getting the feature implemented so that it's usable for me personally. I want the feature, obviously, but I'm not about to implement the featureset across the entire landscape of debug adapter servers. I'm hoping that a single implementation for the client and a single implementation for the server will demonstrate enough to get somebody else to implement a second server/client.

I planned on writing the client version (also called nvim-dap) myself but upon finding this I figured it might be useful to collaborate.

Problem statement: vt200 based terminal interfaces are ubiquitous for the debugging world. node inspect file.js, python -m pdb file.py, gdb ./a.out, lldb ./a.out, etc all use terminal escape sequences that are interpreted via the terminal emulator over PTYs to draw TUIs on the screen. A proper general purpose debugger adapter protocol should account for this.

(If you aren't familiar with the pty/escape sequence/terminal emulator environment then here is a series of articles about how the Microsoft terminal team came to the same conclusions I have. Here is a tutorial series walking you through creating a TUI text editor via these escape sequences.)

e.g. If you go into gnu-nano-dap and open python-pdb-dap it should show you the pdb command prompt in one window, a GUI panel with a play button, a backtrace window, a variable window, a breakpoint window, a source file window. Or, more accurately, some subset of these features dependent on the user.

In the issue linked I'm arguing that this feature should be a standard part of the protocol. If a debugger is willing to show you a TUI command prompt then it should be allowed to.

The plugin nvim-gdb demonstrates the TUI integration idea. However, it doesn't use the DAP and simply scrapes the TUI output to figure out what to do.

[theHamsta]

But you can run almost every debug adpater in a Neovim terminal with support for all escape code

E.g. for debugpy you can choose

                "internalConsole",  -- will print directly in repl-buffer no support for escape sequences, will print to normal vim buffer
                "integratedTerminal",   -- will open and print output directly in Neovim terminal with support of escape sequences.
                "externalTerminal",  -- will open a window with a external terminal (guess not supported by nvim-dap at the moment)

if you want to highlight/conceal escape sequences correctly you'd need to apply special conceal/syntax rule as plugins like this do: https://vimawesome.com/plugin/ansiesc-vim-would-can-one

I don't think it is worth the pain when you can just use a :terminal. You have then a repl with messages from the debug adapter and one terminal with actual program output. The request to run in a actual terminal is standardized: https://microsoft.github.io/debug-adapter-protocol/specification#Reverse_Requests_RunInTerminal (however until recently not supported by lldb-vscode)

The last time I used vscode it didn't support escape sequences in the debug console either, only in integrated terminal.

And yes prompt-buf wasn't well implemented in upstream Vim.

[lanza]

This is unrelated to what I'm talking about. You're talking about functionality for launching the inferior with its' stdin/out/err attached to a specific terminal. I'm talking about the debugger itself.

edit:
Aside from that, you mention neovim specific things. That's explicitly the opposite of what I'm trying to do. The debug adapter server should have 0 knowledge of which IDE it's running with and the debug adapter client should have 0 knowledge of which server it's talking to. There should be a feature check done in the initialization packet and if both sides agree they can handle this extra feature then the PTY is the abstraction layer.

[theHamsta]

It is related because Neovim can only escape terminal sequences natively from output from a program that's its own process. Starting an interior terminal is the Vscode way to get actual formatted output. I've seen that visual studio indeed supports now also escape sequences for the debugger output.

https://code.visualstudio.com/assets/docs/editor/debugging/debugging_hero.png

What you're trying to do would be super nice. I'm just saying that it will be quite an effort to implement all theses escape sequences. Maybe those terminal highlights plugins can understand the most basic ones like color/boldness... Or you set highlights using Lua.

[lanza]

It is related because Neovim can only escape terminal sequences natively from output from a program that's its own process. Starting an interior terminal is the Vscode way to get actual formatted output. I've seen that visual studio indeed supports now also escape sequences for the debugger output.

I'm not sure what you mean by this. neovim uses libvterm and thus has escape sequence processing out the box. I had no intention to implement a terminal emulator, just to do the piping to set them up accordingly.

The most trivial way to do this was to do what mfusseneggar mentioned and just open the dap server via :terminal thedap and have it communicate with the client over tcp. This way the client and server can agree that the 0/1/2 file descriptors are the PTY slave and thus the most trivial approach works.

A step up in difficulty with some hackery would be to just open :term a.out where a.out is just int main() { pause(); } and simply get the device for std{in,out,err} and then pass this filepath to the server and if it's capable of opening a command prompt it can use it. To demonstrate the functionality of hijacking another terminal:

// terminal 1
$ echo 'int main() { pause(); }' > main.c && clang -Wno-implicit-function-declaration && ./a.out

// terminal 2
$ exec 5<>$(lsof -g $(pgrep a.out) | grep ttys | head -1 | awk ' { print $10 }')
$ lldb <&5 1>&5 2>&5

For a real implementation I don't know enough about the neovim APIs off the top of my head but it's doable with libvterm (which is what neovim uses). You could launch the process with stdin/out attached to the DAP communication channels, call posix_openpt and pass the slave in the initialize packet. If the server can use it then it responds with an acceptance and you can then pass the process and pty to libvterm who then uses it to render the terminal emulator.

[mfussenegger]

I overall like the idea of giving users the possibility to use the debug adapters or debuggers CLI directly and have some bi-directional communication with the editor, but I'm not sure how nvim-dap can help or fit into what you're trying to achieve.

Would you want nvim-dap to implement this extended initialization functionality?

I'm hoping that a single implementation for the client and a single implementation for the server will demonstrate enough to get somebody else to implement a second server/client.

If you want a demonstration of it, wouldn't the IP/TCP route be the easiest? You need one debug adapter that implements it and provides the CLI, then you can theoretically use it in any DAP-client. With neovim by using :term, in others by launching the debug adapter in a normal terminal, and then have the DAP-client connect to it.

Extending the protocol to have this part of the initialization protocol, and do a file descriptor handover (if I understand that correctly?) sounds like the more difficult route?

[mfussenegger]

In the haskell language server project there is also a proposal to use dap and the runInTerminal functionality to spawn a REPL: haskell/haskell-language-server#477

The outlined approach also sounds like it should work without protocol extension or extra functionality in clients

[lanza]

I overall like the idea of giving users the possibility to use the debug adapters or debuggers CLI directly and have some bi-directional communication with the editor, but I'm not sure how nvim-dap can help or fit into what you're trying to achieve.

Would you want nvim-dap to implement this extended initialization functionality?

I'm hoping that a single implementation for the client and a single implementation for the server will demonstrate enough to get somebody else to implement a second server/client.

If you want a demonstration of it, wouldn't the IP/TCP route be the easiest? You need one debug adapter that implements it and provides the CLI, then you can theoretically use it in any DAP-client. With neovim by using :term, in others by launching the debug adapter in a normal terminal, and then have the DAP-client connect to it.

Extending the protocol to have this part of the initialization protocol, and do a file descriptor handover (if I understand that correctly?) sounds like the more difficult route?

The reason I brought I this up was the week before I saw your project I started a new neovim lua project called nvim-dap and was planning to implement exactly your project with my feature idea added in. I had like four packets implemented and was writing some horrible lua before I stumbled upon a link on reddit to this project. Thus I figured I'd save my time and check with you guys.

The main benefit of this idea is that it avoids the horrific configuration code necessary to launch debug sessions with anything using the DAP. If you use lldb or gdb, then you know what file a.out, b main, r does. There is no configuration. You launch the debugger and you write those three lines at the prompt like you always do. This VS Code plugin demonstrates the idea... almost. I didn't implement the logic in lldb's lldb-vscode to properly forward the process launched event. (It also only works on macOS as I also didn't build that specific fork of lldb for Linux or Windows).

Istall that plugin (if you have a Mac) and do this:

1. cmd+shift+p to open command prompt
2. select "LLDB: Free Launch"
3. go to the terminal pane that pops up and use lldb normally. If you don't use lldb then you can do this:

(lldb) file someBinaryWithDebugInfoYouBuilt
(lldb) b main
(lldb) r
(lldb) mark

and you've dropped into a full fidelity debug session with almost 0 configuration. In fact, with my plugin from the vscode there is no configuration whatsoever. What you see is a available from a fresh install.

See here. (The mark part is the hack I used mentioned above. It's pretty trivial to forward the launchedProcess event from lldb-vscode over the DAP channel to the client, I just didn't get to it.)

If this feature idea was supported via both the server and client (e.g. the protocol itself supporting it) then the only setup necessary would be:

"debugAdapter": {
  "name": "lldb-vscode",
  "command": "/usr/local/bin/lldb-vscode"
}

And then my Free Launch becomes a generic arbitrary client <-> DAP-server feature. So as soon as SublimeDAP and python-dap (I don't know if these actually exist, I just made them up as a hypothetical) support it, then you could invoke FreeLaunch python-dap from Sublime and then type use the python debugger's command line interface to debug as you always have.