Walking the native stack?

[SamboyCoding]

I'd like to be able to go back in a stacktrace beyond the lowest managed frame (e.g., find what's calling into managed code, in the case of reverse p/invoke). Or for another example, say I have a custom runtime host which loads an assembly and invokes an [UnmanagedCallersOnly] method on it, which in turn calls another method, etc, I'd like to be able to print the stack trace which would include all the managed methods, as a normal Environment.StackTrace would give me, and then also continue down into the unmanaged code of the host - that which called into the managed entry point.

I'm specifically trying on AMD64 architecture right now but would like to have a solution working on x86 eventually.

I've tried:
P/invoking StackWalk64 from dbghelp.dll (having correctly initialized it using SymInitializeW), and I got one frame which was the pinvoke stub, and then garbage data.
Using clrmd's EnumerateStackTrace, which gives me a little more insight into the native pointers of the managed methods, but still stops at the managed entry point.
Calling the RtlCaptureStackBackTrace function (built in to windows) which just returned a whole bunch of 0xBAADF00DBAADF00D pointers, which is obviously some sort of sentinel value.

I then tried a combination of the first two approaches, by setting IncludeContext to true in clrmd, taking the context from the lowest managed frame, converting the span to an equivalent struct and sending this into StackWalk64.
Once again, for the first call this results in correct values for the RIP, RSP, RBP, and RDI registers in the returned context, and their equivalents in the AddrPC and AddrStack members of the Stackframe struct) but the return address is more than an order of magnitude too small (e.g. all executable code is in the 0x7FFXXXXXXXXX range and the return address calculated by the first call to StackWalk64 is in the 0x020XXXXXXXXX range), and as a result any subsequent calls to StackWalk64 use this incorrect ReturnAddress as RIP.

I'm not expecting symbols for the native side of things (i can attempt to symbolicate the trace later), but just a list of pointers would be a good start. Does anyone know the correct way to perform a full thread stack walk of both managed and unmanaged code, as seen in e.g. WinDbg?