Approach for debugging incorrect disassembly location when opening disassembly view

[thomasgoodfellow]

This is when debugging against a RISC-V instruction set simulator that provides a partial GDB Server interface. Generally the debugging works as expected: halts at breakpoints, shows register contents, shows callstack, shows variables. However opening the disassembly view, e.g. from the context menu on the callstack, fails either by showing the disassembly of a different address or simply results in the disassembly window showing "Disassembly not available". From the gdb-remote log it's evident that a memory read is attempted for incorrect addresses, eg:

send packet: $mffff3e4,7dd#fc
read packet: $EFF#D1

where there is no memory mapped at (or near) that address, and no similar address is visible in the GDBServer traffic.

Now I doubt that this is a bug in codelldb or in vscode. My guess is that it stems from the GDB Server implementation of the ISS, quite likely interacting badly with the LLDB library, e.g. by its incomplete support for some query types or GDB remote protocol extensions. So I'd like to walk through the execution of DebugSession:handle_disassemble(), but I don't how to do that with in the vscode/Electron environment. I've stumbled around in the debugger launched by "Toggle Developer Tools" but I haven't found the codelldb source there (perhaps because it's an extension?) to set a breakpoint.Alternatively I think I could add some output messages/messageboxes to that code to see what variable get set to and which pathways are taken, then build that and install from the .vsix?

There is surely some standard approach for debugging something like codelldb but I didn't find a reference for it (of course the multitude of pages on how to use debuggers drown out pages on how to debug the debugger itself). Any and all advice on this most welcome!

[vadimcn]

You can try to enable verbose logging and try to see whether instructionPointerReference's of the stack frames sent in response to stackTrace request make sense.
The address is generated here. Maybe something goes wrong when translating SBAddress to a load address? You should be able to replicate this operation in Python.

[puremourning]

Oops. Did I do something stupid with negative offsets :/

[puremourning]

Yes. Yes I did.

wrapping_add.... should be saturating_add

@thomasgoodfellow you can try a patch like this, though it's certainly incomplete. The math doesn't properly handle the start-of-memory boundary, but I think this might work:

ben@BenMBP2021 CodeLLDB % git diff
diff --git a/adapter/codelldb/src/debug_session.rs b/adapter/codelldb/src/debug_session.rs
index a871d4b..7d04ddb 100644
--- a/adapter/codelldb/src/debug_session.rs
+++ b/adapter/codelldb/src/debug_session.rs
@@ -1214,7 +1214,7 @@ impl DebugSession {
 
         let base_addr = parse_int::parse::<u64>(&args.memory_reference)?;
         let base_addr = match args.offset {
-            Some(offset) => base_addr.wrapping_add(offset as u64),
+            Some(offset) => base_addr.saturating_add(offset as u64),
             None => base_addr,
         };
         let instruction_offset = args.instruction_offset.unwrap_or(0);
@@ -1245,7 +1245,7 @@ impl DebugSession {
         } else {
             let bytes_per_instruction = disassembly::max_instruction_bytes(&self.target);
             let offset_bytes = -instruction_offset * bytes_per_instruction as i64;
-            let start_addr = base_addr.wrapping_sub(offset_bytes as u64);
+            let start_addr = base_addr.saturating_sub(offset_bytes as u64);
             let mut disassemble_bytes = instruction_count * bytes_per_instruction as usize;
 
             let mut dis_instructions = Vec::new();
@@ -1259,6 +1259,9 @@ impl DebugSession {
             }
 
             for shuffle_count in 0..bytes_per_instruction {
+                if start_addr.saturating_sub(shuffle_count) == 0 {
+                    break;
+                }
                 let instructions = disassembly::disassemble_byte_range(
                     start_addr - shuffle_count,
                     disassemble_bytes,
ben@BenMBP2021 CodeLLDB % 

[thomasgoodfellow]

Thanks for all those ideas & apologies for the belated response. Although the address wrap-around quite plausibly is a general issue I don't think it's the direct cause here. The ISS is providing a bare-metal device (no OS, no loader) with the .text segments from 0x10000000 (i.e. ROM) and .data segments from 0x20000000 (i.e. RAM). So only a huge offset from any code address could lead to a wrap-around and there is no readable memory in the first 256MB. Also I did try patching with the saturating math operations and the behavior was unchanged.

I've tried adding some logging through the info! macro but didn't see it in the LLDB output or debug console, so either that gets logged elsewhere or I've got the wrong filter level set - continuing to experiment with this.

(I also tried the old-school approach of inserting int3 instructions in the hopes of getting a trap to start debugging from that point, but that simply broke the debugging session without any obvious diagnostic)

[vadimcn]

I think I've fixed this in v1.11.5, please try it.

[thomasgoodfellow]

Thanks very much for that, the fix works well (rather more elegant than the kludge I'd hacked up) - and apologies that I didn't get back to testing sooner). Also thanks for this tool - I hadn't thought to look to vscode for bare-metal debugging but so far it's impressive and codelldb is well-documented and has a good set of options.

[puremourning]

There is surely some standard approach for debugging something like codelldb

I just attach to the process with... you guessed it, codelldb.

[thomasgoodfellow]

That sounds great, but could you please elaborate - this would mean running another instance of VSCode with a debug configuration for attaching, along the lines of https://stackoverflow.com/a/60440766? And then would simply opening the codelldb source allow the setting of breakpoints in the extension?

[puremourning]

yes, exactly. I don't use vscode but the principle is the same: attach to the codelldb process from another session.

[thomasgoodfellow]

So I got to debugging codelldb with cgdb and it did indeed go well; only issue I had was dangling code and codelldb processes. Thanks for the help!

[thomasgoodfellow]

Answering the original question of "how to debug CodeLLDB", I just hadn't read the documentation well enough:

1. Build codelldb for debugging, per building.md: make dev_debugging
2. Launch vscode with that build: code --extensionDevelopmentPath=codelldb-build-path
3. Start debugging the target code (e.g. the ISS in my case), vscode halts at the start
4. Attach a native debugger to the codelldb process, e.g. ps -u thomasg f | grep codelldb to find the PID then cgdb -p PID (usually see two; the one of interest has "--settings" in the command line)
5. Set breakpoints, e.g. in handle_disassemble(), continue, then interaction in vscode hits breakpoints with source & debug info
6. code and codelldb processes get left dangling, especially when halting attached debugger, need to killall them to keep it clear which one to attach to.