List of delinker/unlinker tooling projects

[boricj]

This thread is for cataloging tools that can delink (or unlink) code.

Self-reported:

• boricj's ghidra-delinker-extension
• boricj's ghidra-unlinker-scripts (NOTE: obsoleted by ghidra-delinker-extension)

Reported by @widberg (#1 (comment)):

• endrazine's Witchcraft Compiler Collection
  • DEF CON 24 talk
• jonwil's unlinker
  • Announcement thread on forum.exetools.com
• jonwil's unlinkerida
  • Announcement thread on forum.exetools.com

Reported on Anda Pirate's Discord server:

• jnider's delinker

[encounter]

More specialized, but "unlinkers" nonetheless:

• encounter/decomp-toolkit: Specifically for GameCube and Wii (PowerPC), analyzes and splits game binaries into relocatable objects. Powers many matching decompilation projects.
• ethteck/splat: For N64, PSX, PS2 and PSP (MIPS). Also powers many matching decompilation projects. Emits assembly instead of writing relocatable objects directly, but that still counts, in my opinion.

Colloquially, the game decompilation community calls this process "splitting".

[boricj]

Interesting. I suspected there was some tooling out there, but I rejected perfect decompilation and every project out there seemed to go for it. I assumed I was off the beaten path and decided to do my own thing based on Ghidra, without looking at prior art.

My approach/implementation is too different for me to make comparisons after a cursory glance at these two repositories. It does appear that my Ghidra extension is focused on just one problem, but managed to solve it in a more general way.

Is there a forum/Discord server/something else for this sort of thing? I'm way out of the loop, but my distinctive take on this topic and my experience with it might be of interest to others.

[encounter]

Matching/perfect decompilation is hard, so I get it, but the results are quite cool! Tooling for non-matching decompilation (e.g. verifying functional equivalency without completely matching output) needs more exploration. (Maybe Ghidra P-Code could be of use?)

I think the Ghidra extension approach seems quite cool. Our approaches do differ quite a bit. For our projects, we focus on:

• Reproducibility: We don't want to distribute the game binary (or portions of it), so we rely on the user to provide it. The tooling uses a configuration file to inform the splitting process.
• Accuracy: We can get much better results by performing our own relocation analysis than relying on a more general tool like Ghidra, since we target a small number of compilers and compiler versions. Sometimes details as small as padding between objects can tell us where translation units start and end, and clever heuristics can eliminate a lot of manual work for the user.

This kind of work has been a focus of mine for a few years now, so I could ramble on 🙂

One of the more central communities is decomp.me (Discord), a site for collaborating on decompilation matches. This extension was mentioned a couple times in there. There's also the GC/Wii Decompilation Discord which has channels for the decomp-toolkit and objdiff projects.

[punpckhdq]

This is unpublished work, but for the past two years I have been writing on and off a standalone object splitter tool for x86 PE executables paired with debug .pdb archives.

I intend to rewrite this framework in order to support multiple architectures and an interactive binary diff/split interface, and then release it as open source software.

[boricj]

Looks neat. It seems like I've tackled this topic in a different way than the rest of the game decompilation community.

I will say that in my case, supporting multiple ISAs and object file formats in a decoupled manner is quite the challenge. I've managed to find a data model and an architecture that work mostly OK, but I still have a bunch of refactorings and clean-ups to do before I'll be happy with it. Also, this extension likes to silently break if given the chance, I've invested a lot into regression testing and it's barely enough to stop it from imploding.

Feel free to update your comment with a link whenever you release it.

[kotcrab]

Inspired by your extension I implemented object file exporter for PSP kernel modules in the ghidra-allegrex plugin, the usage guide is here. The main goal here is help with debugging reverse engineered modules. By mixing reverse engineered code with the original implementation we can use this to find functions with RE errors.

Those modules always have relocation tables but it's in a different format than standard relocations (e.g. HI/LO relocations don't have to follow each other in the table) so the biggest challenge here is converting them to standard format that our linker can understand. For this I used output from the decompiler which seems to work mostly okay but it has its own issue as it's not always clear how to process the resulting pcode.

[boricj]

Cool! I've had a glance at the exporter, it seems fit for purpose. Being able to leverage the original relocation table simplifies the problem a fair bit, but relative relocations internal to a section won't appear in it. That's probably fine on MIPS though because PC-relative branches usually do not span across functions in that architecture. I haven't experimented with the decompiler's output for delinking purposes, I guess PSP kernel code is probably less horrible than shitty PS1 game code to the point where that's a viable option.

I can't seem to find any details about that particular flavor of ELF you're exporting to, but it seems just special enough that a general-purpose ELF object file exporter like mine maybe can't quite accommodate it. That being said, I've found that there's a large discrepancy between what the ELF specification for MIPS says and what MIPS toolchains do ; I've yet to eliminate all of the wrong assumptions caused by the former on my side (like the undocumented GNU extensions around HI16/LO16 that I've somehow managed to misidentify as an assembler peephole optimization and successfully worked around with the Branch instruction target adjusted to deoptimize possible HI16 relocation inside delay slot nonsense, I really gotta get rid of that).

[kotcrab]

PC-relative branches usually do not span across functions in that architecture

Yeah I don't think branches can span across functions in normal compiled code. And true they are relative to PC so no relocation entry for them. Kernel modules are also tiny compared to games so it's easier to verify output for the whole thing.

What I exactly use decompiler output for: I start from LO16 relocation and then I try to find all HI16 relocation that are linked with this LO16 relocation, this also gives me the symbol address. Maybe this approach could be useful for you. I then also do some verification against the original table to see if any HI16 were not found or if one HI16 relocation somehow matched to multiple LO16, that would indicate some error.

I don't think there's anything too special about the output ELF format, maybe other than the non-standard section type PSP uses for the relocation table. Some flag values, etc. I just made the same to match what the compiler normally generates. I thought about adding it to your extension but since it's dealing with relocations that are unique to PSP in the end I decided to just keep it all in the ghidra-allegrex extension, that gives some code reuse too.

For PSP what's cool we can look at the kernel module responsible for applying the relocations and from that know what types are supported and how they are applied (there isn't that many possible types). One interesting quirk is that the HI16 relocation doesn't have to be completed by LO16, the loader will use any first non HI16 relocation in the table. It can happen in practice though must be very rare.