Add support for debugging PennyLane (Python) and Catalyst (C++) simultaneously #1712

mlxd · 2025-05-02T20:59:06Z

Context: Debugging multi-language programming environments can be a challenge due to the intermingling and swapping between all participating languages. In the case of PennyLane and Catalyst, this is a mix of Python and C++, through our own libraries all the way through LLVM and back. Interactive debugging of a multi-language environment can require some custom setup to ensure the environment allows for useful interactive data inspection and validation.

VSCode provides support for mixed-mode debugging where a single editor can support (through the Python and C++ debugging extensions) interactive breakpointing, stepping, and inspection of the running program, where the view can switch between the Python debugger and the C++ debugger, once running. This PR adds preliminary support for this mixed-mode debugging, allowing the user to set breakpoints in their PennyLane program their Catalyst (+/- LLVM) internals, and step all the way through to investigate all aforementioned stages.

Some notes:

Use of this functionality requires building Catalyst with debug symbols. This can be achieved via make all BUILD_TYPE="RelWithDebInfo", and may generated approximately 80GB of libraries and binaries. Added internal (BUILD_TYPE) and external (BUILD_TYPE_EXT) configuration for build-flags to restrict debug symbols to our libs and bins only.
Launching the C++ debugger requires attaching to a running process. This requires sudo privileges.
The spawned compiler subprocess requires receipt of a SIGCONT to continue executing after the C++ debugger has attached.
~~Setting the active qml.qjit(debug_compiler=True) option will not enable the support unless the Python process is called from a debugger (pdb, debugby, etc)~~ This restriction has been removed.

Description of the Change: As above.

Benefits: Enables debugging of mixed mode (Python + C++/LLVM) programs directly through VSCode.

Possible Drawbacks:

Related GitHub Issues:

…yst into add_dualmode_debugging

dime10 · 2025-05-02T22:35:22Z

Use of this functionality requires building Catalyst with debug symbols. This can be achieved via make all BUILD_TYPE="RelWithDebInfo", and may generated approximately 80GB of libraries and binaries.

To mitigate one could avoid building llvm itself with debug symbols, the catalyst code base itself is not that bad in debug mode 🤔

frontend/catalyst/debug/__init__.py

dime10

This is awesome!!! Thanks Lee :)

.vscode/launch.json

frontend/catalyst/compiler.py

mlxd · 2025-05-02T23:24:22Z

Use of this functionality requires building Catalyst with debug symbols. This can be achieved via make all BUILD_TYPE="RelWithDebInfo", and may generated approximately 80GB of libraries and binaries.

To mitigate one could avoid building llvm itself with debug symbols, the catalyst code base itself is not that bad in debug mode 🤔

This is a good point; I may try to restrict the debug symbols to the Catalyst additions and see what savings we get.

mlxd · 2025-05-30T19:31:24Z

Use of this functionality requires building Catalyst with debug symbols. This can be achieved via make all BUILD_TYPE="RelWithDebInfo", and may generated approximately 80GB of libraries and binaries.

To mitigate one could avoid building llvm itself with debug symbols, the catalyst code base itself is not that bad in debug mode 🤔

This is a good point; I may try to restrict the debug symbols to the Catalyst additions and see what savings we get.

This works, so extended the Makefile to offer BUILD_TYPE for our stuff, and BUILD_TYPE_EXT for all external packages. that way we can control both as needed.

dime10

Thanks Lee, I would like to test it before approving, but I'm having some difficulties 😅

mlir/Makefile

dime10 · 2025-05-30T20:48:16Z

frontend/catalyst/debug/debugger.py

+import sys
+
+
+def is_debugger_active() -> bool:


How is this function useful, if we don't use it internally?

Looking at this function again, I don't think we should add it to the Catalyst API, it's completely unrelated to Catalyst code.

dime10 · 2025-05-30T20:54:05Z

doc/dev/debugging.rst

+
+    {
+        "python.defaultInterpreterPath": "${env:VIRTUAL_ENV}",
+        "python.terminal.launchArgs": [],
+    }


Could you explain these options (and why they are needed)?

Sure, will add some references and guidelines on how https://code.visualstudio.com/docs/configure/settings are used here with the debugger.

dime10 · 2025-05-30T20:56:42Z

frontend/catalyst/compiler.py

+                        f"""Ensure C++ debugger is attached and running before continuing with:
+                        kill -s SIGCONT {p.pid}"""
+                    )
+                    p.send_signal(signal.SIGSTOP)


Do we know the process hasn't done any work yet by the time we get here?

It's a good question. The assumption (from most places I've read) is that this should be (reasonably) fine. But if we 100% need to be strict on it, we can modify the process launch options to ensure the signal is the first thing that is hit.

Is it just based on hoping that the OS doesn't schedule the subprocess thread until we hit this statement, or is there some other reason it would hold? For instance, does the process only start running once we call p.communicate (not sure how it works)?

So communicate is something akin to a synchronisation point for the child-process --- it allows data to be sent to the process (assuming it is waiting for it) and receives data from the process.

For a quick 'n' dirty example of whether it is fast enough to ensure the process isn't running anything, we can opt for something akin to https://stackoverflow.com/questions/50002804/create-subprocess-in-python-in-suspended-state

In practice, use of the preexec_fn argument in Popen should work just fine here (as in, we create the PID, and before execution happens we immediately put it into the suspend state). This should be fine. Though, if we are ok with letting the child process set itself up first, chances are quite low anything will execute. Taking a simple compiled C binary that outputs a string test, we can try out the following to see when the output hits the screen (creating a file should work too):

import os import signal def f(): "pre-execute function to print the new process PID" print(os.getpid()) print("Starting process") p = Popen(["/tmp/test"], preexec_fn=f) print("Running process") # Stop process after creating the Process and a print call. # If the `/tmp/test` binary can output before this line is hit, # we need to stop within the preexec_fn. Otherwise, all good # to set up the process. p.send_signal(signal.SIGSTOP) # Have some wait-time that we control i = input("enter/return to continue") # Allow direct input/output from communicate p.communicate() # It should be done by now print("Finished process") # explicitly kill the process p.kill() print("Killed process")

Reasoning for favouring not relying on the preexec_fn is there appears to be some (valid) ongoing attempts to remove this from CPython (see python/cpython#82616).

Thanks for the info! I'll leave it up to you then if we want to use preexec or not, although in the current approach maybe we could at least send the signal before printing? 😅

Fine by me. We can always revert if the above CPython changes make it to a release.

Update: As an aside, since the printing will happen from the subprocess, we may have issues with capturing the stdout and stderr for reporting. I'll see if this is something that can be mitigated, but if not, we just use the existing.

dime10 · 2025-05-30T21:03:51Z

frontend/catalyst/jit.py

@@ -96,6 +96,7 @@ def qjit(
    circuit_transform_pipeline=None,
    pass_plugins=None,
    dialect_plugins=None,
+    debug_compiler=False


Personally I would have preferred not adding another keyword to this decorator, but it's not the end of the world 😌
The automatic detection based on the already active Python compiler session was actually really neat in that way.

Yea, i'd agree it was nice -- maybe we can chat next week and see which seems better. This way allows attaching from a non-VSCode/non-Python debug session, so if people prefer to use it without, they can. But happy to defer either way.

Good point. I was able to get it running now and the process is pretty straightforward, so I'm happy to go this route if you prefer :)

Although, I just thought of another alternative. If we want to contain this sort of functionality to the debug module, we could add a context manager for emitting the signal, similar to the instrumentation one. A bit more inline with what we already have, and keeps it out of the regular user options. Just an idea though.

Actually, not a bad idea. Let me see what is needed to try that approach. If it becomes a pain, we can always support that in a follow-up, and leave this as usable in the current form for now.

doc/dev/debugging.rst

Co-authored-by: David Ittah <dime10@users.noreply.github.com>

codecov · 2025-05-30T22:22:53Z

Codecov Report

Attention: Patch coverage is 65.00000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 96.53%. Comparing base (f484b02) to head (634f8fd).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
frontend/catalyst/compiler.py	60.00%	4 Missing and 2 partials ⚠️
frontend/catalyst/debug/debugger.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1712      +/-   ##
==========================================
- Coverage   96.60%   96.53%   -0.08%     
==========================================
  Files          82       83       +1     
  Lines        9211     9226      +15     
  Branches      872      875       +3     
==========================================
+ Hits         8898     8906       +8     
- Misses        254      259       +5     
- Partials       59       61       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

doc/dev/debugging.rst

dime10 · 2025-06-02T18:28:07Z

doc/dev/debugging.rst

+                "request": "attach",
+                "program": "${command:python.interpreterPath}",
+                "processId": "${command:pickProcess}",
+                "MIMode": "gdb",


Mentioned this in the internal guide, but with the given configuration I get

Unable to start debugging. Launch options string provided by the project system is invalid. Unable to determine path to debugger. Please specify the "MIDebuggerPath" option.

Not sure what I would put for the path though since I can't find gdb on my mac. "MIMode": "lldb" works though.

Yea, as in the other, I think we need some form of pre-execution script to define a system-dependent argument for this using a custom task

For a basic solution, I think it's ok to mention in the instructions to switch to lldb if on mac.

dime10 · 2025-06-02T18:31:55Z

doc/dev/debugging.rst

+                "name": "(C++): Attach To Executing Python Process",
+                "type": "cppdbg",
+                "request": "attach",
+                "program": "${command:python.interpreterPath}",


I'm confused by this parameter. I tried removing it but then VSCode complains that "program" is not specified. The thing is though, the process we are attaching to is not running Python, it is running the catalyst program (the compiler). Further, the value really doesn't seem to matter as long it's a valid path because I can put /bin/echo and it still works 😅

Does it influence where the source directory is parsed for the debug information?
If not, all good to put in a no-op here, just to please the VSCode integrator.

Does it influence where the source directory is parsed for the debug information?

Hmm, how can I check this? I used the echo path and it still stopped on the breakpoint in my local catalyst installation.

dime10 · 2025-06-02T18:36:52Z

frontend/catalyst/compiler.py

+                        f"""Ensure C++ debugger is attached and running before continuing with:
+                        kill -s SIGCONT {p.pid}"""
+                    )
+                    p.send_signal(signal.SIGSTOP)


Is it just based on hoping that the OS doesn't schedule the subprocess thread until we hit this statement, or is there some other reason it would hold? For instance, does the process only start running once we call p.communicate (not sure how it works)?

dime10 · 2025-06-02T18:41:33Z

frontend/catalyst/jit.py

@@ -96,6 +96,7 @@ def qjit(
    circuit_transform_pipeline=None,
    pass_plugins=None,
    dialect_plugins=None,
+    debug_compiler=False


Good point. I was able to get it running now and the process is pretty straightforward, so I'm happy to go this route if you prefer :)

Although, I just thought of another alternative. If we want to contain this sort of functionality to the debug module, we could add a context manager for emitting the signal, similar to the instrumentation one. A bit more inline with what we already have, and keeps it out of the regular user options. Just an idea though.

Co-authored-by: David Ittah <dime10@users.noreply.github.com>

mlxd added 4 commits May 2, 2025 15:38

Initial support for active debugger session for compiler attachment

e28ffbe

Add debug flag autocheck

52d848b

Ensure VSCode debugger launch file is tracked

6101c6f

Add debug_compiler args to CompileOptions and qjit

4d1ff94

mlxd changed the title ~~Add support for debugging PennyLane (Python) and Catalyst (C++) simultaenously~~ Add support for debugging PennyLane (Python) and Catalyst (C++) simultaneously May 2, 2025

mlxd and others added 4 commits May 2, 2025 17:13

Fix stdout stderr from popen

21686dd

Merge branch 'main' into add_dualmode_debugging

dfd5007

Update debugger prompt

97cd18b

Merge branch 'add_dualmode_debugging' of github.com:PennyLaneAI/catal…

44fbf63

…yst into add_dualmode_debugging

dime10 reviewed May 2, 2025

View reviewed changes

frontend/catalyst/debug/__init__.py Outdated Show resolved Hide resolved

dime10 reviewed May 2, 2025

View reviewed changes

.vscode/launch.json Outdated Show resolved Hide resolved

frontend/catalyst/compiler.py Outdated Show resolved Hide resolved

mlxd and others added 8 commits May 6, 2025 04:37

Merge branch 'main' into add_dualmode_debugging

e6b4849

Merge branch 'main' into add_dualmode_debugging

a56a45c

Add external build type makefile arg for middle-end

6d77976

Add debugging docs for mixed-mode

afba2ce

Remove restriction on launching from a Python debugger

deb0502

Add caption for filenames

23a8cc5

Add python debug session checker

a6e7514

Use editor interpreter path

fcc3895

Move debug config to docs

f40a06b

mlxd marked this pull request as ready for review May 30, 2025 19:33

mlxd added 3 commits May 30, 2025 15:38

Add CL

9c84fcb

Rehide the internal active debug check

dbe0908

fix format

25583b3

mlxd requested a review from dime10 May 30, 2025 19:44

dime10 reviewed May 30, 2025

View reviewed changes

Ensure stdout,stderr tracked from subprocess

fa50232

Update doc/dev/debugging.rst

004ebff

Co-authored-by: David Ittah <dime10@users.noreply.github.com>

dime10 reviewed Jun 2, 2025

View reviewed changes

mlxd and others added 2 commits June 2, 2025 16:12

Update doc/dev/debugging.rst

6ddad2a

Co-authored-by: David Ittah <dime10@users.noreply.github.com>

Update doc/dev/debugging.rst

634f8fd

Co-authored-by: David Ittah <dime10@users.noreply.github.com>

Add support for debugging PennyLane (Python) and Catalyst (C++) simultaneously #1712

Are you sure you want to change the base?

Add support for debugging PennyLane (Python) and Catalyst (C++) simultaneously #1712

Uh oh!

Conversation

mlxd commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dime10 commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dime10 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mlxd commented May 2, 2025

Uh oh!

mlxd commented May 30, 2025

Uh oh!

dime10 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlxd May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dime10 Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlxd Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dime10 Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dime10 Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mlxd commented May 2, 2025 •

edited

Loading

dime10 commented May 2, 2025 •

edited

Loading

mlxd May 30, 2025 •

edited

Loading

dime10 Jun 2, 2025 •

edited

Loading

mlxd Jun 3, 2025 •

edited

Loading

codecov bot commented May 30, 2025 •

edited

Loading

dime10 Jun 2, 2025 •

edited

Loading

dime10 Jun 2, 2025 •

edited

Loading

dime10 Jun 2, 2025 •

edited

Loading