Notes - Trying to run go binaries on Airport

[krackers]

Unfortunately I wasn't really able to get it to work. But here's some notes in case someone else wants to pick up where I left off.

First thank you to @gchehab for providing the precompiled BSD base sets (#4) which makes spelunking a lot easier. It includes gcc so while it's possible to compile newer versions of curl, perl, rsync, etc. directly on device, this is a bit of a pain and it's probably quite slow to do so as well. Go has legendary cross-compile abilities (since it basically avoids libc entirely and makes direct syscalls), and it natively supports arm v6 + netbsd. Plus binaries are statically linked by default. So I thought this would just work out of the box.

Issue 1: Newer versions of Go use a version of lwp_park syscall not supported on netbsd 6. This is described in https://mail-index.netbsd.org/tech-pkg/2018/07/05/msg019971.html – You can either use go1.9, or just revert the aforementioned commit.

Issue 2: While that's sufficient to have the resulting binary begin the go runtime initialization, it gets stuck after the call to lwp_park (semasleep). It seems that there's a deadlock somewhere, nothing is waking it up (semawakeup). Tracing with gdb shows that there's only one thread active, so something about the thread creation seems wonky.

I also tried using the most modern go version with the above commit reverted. This crashes even before the call to lwp_park, on the thread creation itself which returns EINVAL. After some debugging, seems something is weird with the lwp_mcontext_init call, it sets R15 (PC) to the address to resume from (lwp_tramp), but for some reason the alignment of the function seems to be off and bsd doesn't like it. I didn't really debug further, I just brute-forced by inserting nop until the alignment worked out and it got past this. Unfortunately it still hung on the lwp_park.

The hang there might be a kernel issue, I see netbsd had some kernel changes to support Go (Go is also unique in that it makes syscalls directly which I recall is not really supported on BSDs, so I can imagine that it's doing something which bsd doesn't like).

If anyone wants to give this a shot, then trying to see why the lwp_create doesn't actually create a new thread is probably the way to go. I suspect that this created thread is what would send the wakeup call and get things moving.

[krackers]

Ok it seems that the only reason 1.9 seemed to get further was because it was checking the return code from lwp_create incorrectly. It returns EINVAL there as well, but it wasn't properly negating it before deciding to error so it silently assumed it succeeded. This explains why there was no second thread being created.

So now the main thing to answer is why exactly lwp_create returns EINVAL for the value of r15 (pc) we try to set to lwp_tramp. Looking at netbsd source the only possible relevant thing seems to be cpu_mcontext_validate, but printing out the proposed pc it seems to pass that fine.

Edit: One one of my attempts that involved manually inserting Noops before lwp_tramp in order to force the lower 4 bits of the address to all be zeros, things made quite a bit of good progress: the runtime was able to spawn up to 4 goroutines. Unfortunately it then seems to get corruption or something, since one of the PCs gets set to an address which is invalid. I'm not sure where this corruption is coming from.

gdb also seems to not be able to show all the threads properly.

[krackers]

The address alignment is actually even more specific. Bits 6 and 7 (zero-indexed) also seem to need to be 0, and bit 5 seems to need to be 1. So the address should be like 0bXXXX0010000. No idea why this is, I can't find an explanation from looking at netbsd source, and it evidently isn't true on other netbsd platforms.

Fwiw I did try doing the same alignment trick with go 1.10 binary, hoping the go runtime might be different in a good way, but I still see the same goroutine error with random PC. I'm out of ideas now.

[gchehab]

Hey @krackers, great work! Never tried to mess with go, but it would be great to cross-compile Rclone to the airport, I will surely take a try on that once I get back home from the holidays. Dont remember messing with address alignment, however, I might have read something bout different arm versions that have some tricks on alignment.