FastAPI programmatic backend: execution model?

[wlach]

Am I correct in thinking that the FastAPI Programmatic Backend runs the Marimo kernel in the same process as the FastAPI service? If that's the case, it seems likely a long-running non-async operation could block the server from servicing other requests due to the GIL?

E.g. something like this:

import marimo

__generated_with = "0.14.13"
app = marimo.App(width="medium")


@app.cell
def _():
    import marimo as mo
    import time
    return mo, time


@app.function
def find_prime_inefficiently(n):
    # This function uses a slow, pure-Python loop
    # and holds the GIL for the entire duration.
    candidate = 2
    count = 0
    while count < n:
        is_prime = True
        for i in range(2, int(candidate**0.5) + 1):
            if candidate % i == 0:
                is_prime = False
                break
        if is_prime:
            count += 1
        candidate += 1
    return candidate - 1


@app.cell
def _(mo):
    n_slider = mo.ui.slider(1, 100000000, value=1000, label="Find the n-th prime")
    n_slider
    return (n_slider,)


@app.cell
def _(mo, n_slider, time):
    start = time.time()
    result = find_prime_inefficiently(n_slider.value)
    end = time.time()
    mo.md(f"The {n_slider.value}-th prime is {result}. It took {end - start:.2f} seconds.")
    return


@app.cell
def _():
    return


if __name__ == "__main__":
    app.run()

If this is the case, are there any mitigations or workarounds? Is moving computation into another process on the table as a possible improvement?

[akshayka]

Yes you're correct. Each kernel runs in its own thread, but they all run in the same process (and in the same process as the server). Although won't Python release the GIL between instructions at its own discretion?

Do you want each session ("kernel") to run in its own process (that's what we do in edit mode)? For app mode that overhead felt overkill (and led to memory pressure). But we could potentially make it a flag.

[wlach]

Yes you're correct. Each kernel runs in its own thread, but they all run in the same process (and in the same process as the server). Although won't Python release the GIL between instructions at its own discretion?

I wasn't sure, I was under the impression that a single thread could still hog the GIL. Doing some more research it sounds like (at least with any version of Python that's at all recent) there's some guarantee that other threads will get some time. Although it sounds like a loop like mine would make overall performance pretty bad. I admit I haven't done any empirical testing on this though.

Do you want each session ("kernel") to run in its own process (that's what we do in edit mode)? For app mode that overhead felt overkill (and led to memory pressure). But we could potentially make it a flag.

It might be nice to have the option, although I don't have a strong opinion just now. Probably worth documenting the behaviour at the very least so people aren't caught off guard? Another gotcha I found is that the FastAPI server needs sticky sessions enabled on your server (since requests for a session in another process will fail) and that you can only run one uvicorn process at a time (although maybe there's a way around that if you could make the sticky session apply to a specific process... not sure).

[akshayka]

I wasn't sure, I was under the impression that a single thread could still hog the GIL. Doing some more research it sounds like (at least with any version of Python that's at all recent) there's some guarantee that other threads will get some time.

Yes, they will get some time. Each kernel in marimo has a few threads running concurrently with the main thread:

1. a thread that forwards stdout/stderr to the frontend even while user code is executed
2. a thread that services code completion requests
3. a thread that watches for changes to modules (only relevant in edit mode)

But yes a very tight (Python) loop would probably degrade performance.

It might be nice to have the option, although I don't have a strong opinion just now. Probably worth documenting the behaviour at the very least so people aren't caught off guard?

Makes sense

Another gotcha I found is that the FastAPI server needs sticky sessions enabled on your server (since requests for a session in another process will fail)

Would you be open to making a docs PR for this?

Thanks for the discussion by the way!

[wlach]

Yup, I can make a PR. Might have to wait a week but I'll see if I can put something together.