Questions about how to manage Python ThreadState

[AlexanderWells-diamond]

I'm interested in understanding how the Greenlet library constructs and uses new frames. I maintain a library that does a very similar thing, but we recently discovered that we are apparently incompatible with Python3.11 (but strangely not 3.12 or 3.13...).

Prior to Python3.11, we had been calling PyThreadState_GET(), saving the returned structure, and then modifying it to act as a base for newly executed Python code. You can then see in the two linked functions the cleanup of the edited frame and the restoration of the saved frame.

The issue began when Python3.11 removed the "frame" member of the PyThreadState structure. I thought that all I needed to do was to swap to using the PyThreadState_New, PyThreadState_Swap, and PyThreadState_Delete calls to create a new threadstate, substitute it in, and then when work is complete to just swap it back out and delete the newly created one. That can be seen in the #if sections of this function. This had appeared to be fine for a couple of years, but recently we found it apparently was incompatible when trying to run alongside PyQt - we see an unexplained segfault on the transition between C and Python layers.

Unfortunately, it also seems this usage of these APIs is invalid in Python3.11, at least according to this error when running using PyDebug. Curiously this error is removed in Python3.12 and 3.13, and indeed we do not see any issue (so far) in those versions when running using the New, Swap, and Delete mechanism. So for now this issue appears to be restricted to Python 3.11.

tl;dr:

I've been trying to understand how Greenlet does its Python frame handling. I note it does not use anything except the PyThreadState_Get API, and all other access to the PyThreadState structure appears to be direct. I found the thread saving code and restoring code, but am struggling to see how the threadstate itself is being modified to allow new code to be run. Does that happen down in the assembly code?

Apologies for the long post, and thank you in advance for any pointers!

[jamadden]

I think we're doing different things. greenlet isn't creating and destroying new thread states, it's simply copying and restoring some things from the existing state. There are no public APIs for this, so greenlet has to have (version-specific!) custom code that accesses CPython's internals. The assembly code has nothing to do with CPython, it's operating on the machine (C) stack.