How to properly stop uvicorn to not produce zombie python process?

[piotrekkr]

On my local dev env (inside python docker container) I'm trying to use supervisord with an uvicorn program.

[supervisord]
loglevel=warn
logfile=/proc/self/fd/1
logfile_maxbytes = 0
nodaemon=true
pidfile=/tmp/supervisord.pid

[unix_http_server]
file=/tmp/supervisor.sock
username = dummy
password = dummy

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock
username = dummy
password = dummy

[program:api]
command=uvicorn aiapi.main:app --host 0.0.0.0 --port 8080
    --log-config src/logging.conf
    --reload --reload-dir src
    --timeout-graceful-shutdown 0
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stdout
stderr_logfile_maxbytes=0
loglevel=debug
stopsignal=INT
startsecs=0
stopwaitsecs=60

When I start supervisord with: supervisord -c .docker/supervisor/dev.conf I get uvicorn process up and running with some additional watcher processes

⬢ [Docker] ❯ supervisord -c .docker/supervisor/dev.conf
{"message": "Will watch for changes in these directories: ['/app/src']", "asctime": "2025-04-15 08:54:39,989", "levelname": "INFO", "name": "uvicorn.error", "module": "config", "filename": "config.py", "funcName": "__init__", "lineno": 318, "timestamp": "2025-04-15T08:54:39.989084+00:00", "level": "INFO"}
{"message": "Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)", "asctime": "2025-04-15 08:54:39,989", "levelname": "INFO", "name": "uvicorn.error", "module": "config", "filename": "config.py", "funcName": "bind_socket", "lineno": 524, "color_message": "Uvicorn running on \u001b[1m%s://%s:%d\u001b[0m (Press CTRL+C to quit)", "timestamp": "2025-04-15T08:54:39.989300+00:00", "level": "INFO"}
{"message": "Started reloader process [294] using WatchFiles", "asctime": "2025-04-15 08:54:39,989", "levelname": "INFO", "name": "uvicorn.error", "module": "basereload", "filename": "basereload.py", "funcName": "startup", "lineno": 80, "color_message": "Started reloader process [\u001b[36m\u001b[1m294\u001b[0m] using \u001b[36m\u001b[1mWatchFiles\u001b[0m", "timestamp": "2025-04-15T08:54:39.989437+00:00", "level": "INFO"}
{"message": "Started server process [296]", "asctime": "2025-04-15 08:54:45,284", "levelname": "INFO", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "_serve", "lineno": 83, "color_message": "Started server process [\u001b[36m%d\u001b[0m]", "timestamp": "2025-04-15T08:54:45.284963+00:00", "level": "INFO"}
{"message": "Waiting for application startup.", "asctime": "2025-04-15 08:54:45,285", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "startup", "lineno": 48, "timestamp": "2025-04-15T08:54:45.285242+00:00", "level": "INFO"}
{"message": "Application startup complete.", "asctime": "2025-04-15 08:54:45,285", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "startup", "lineno": 62, "timestamp": "2025-04-15T08:54:45.285408+00:00", "level": "INFO"}

And process list looks like this:

⬢ [Docker] ❯ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
app            1  0.0  0.0   2940  1408 ?        Ss   08:54   0:00 tail -f /dev/null
app           47  0.1  0.0   6420  4736 pts/0    Ss   08:54   0:00 zsh
app          239  0.3  0.0   6740  5116 pts/1    Ss   08:54   0:00 zsh
app          293  0.8  0.1  40000 34648 pts/0    S+   08:54   0:00 /opt/venvs/app/bin/python /opt/venvs/app/bin/supervisord -c .docker/supervisor/dev.conf
app          294  0.9  0.1 108144 35376 pts/0    Sl   08:54   0:00 /opt/venvs/app/bin/python /opt/venvs/app/bin/uvicorn aiapi.main:app --host 0.0.0.0 --port 8080 --log-config src/logging.conf --reload --reload-dir src --timeout-graceful-shutdown 0
app          295  0.1  0.0  20736 16728 pts/0    S    08:54   0:00 /opt/venvs/app/bin/python -B -c from multiprocessing.resource_tracker import main;main(4)
app          296 10.5  0.5 558180 184344 pts/0   Sl   08:54   0:06 /opt/venvs/app/bin/python -B -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=5, pipe_handle=7) --multiprocessing-fork
app          361  0.0  0.0   8484  4352 pts/1    R+   08:55   0:00 ps aux

Then I use CTRL+C to stop supervisor and it stops all programs:

^C2025-04-15 08:56:03,967 WARN received SIGINT indicating exit request
2025-04-15 08:56:03,967 WARN received SIGINT indicating exit request
{"message": "Shutting down", "asctime": "2025-04-15 08:56:04,052", "levelname": "INFO", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "shutdown", "lineno": 263, "timestamp": "2025-04-15T08:56:04.052105+00:00", "level": "INFO"}
{"message": "Cancel 0 running task(s), timeout graceful shutdown exceeded", "asctime": "2025-04-15 08:56:04,152", "levelname": "ERROR", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "shutdown", "lineno": 283, "timestamp": "2025-04-15T08:56:04.152764+00:00", "level": "ERROR"}
{"message": "Waiting for application shutdown.", "asctime": "2025-04-15 08:56:04,153", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "shutdown", "lineno": 67, "timestamp": "2025-04-15T08:56:04.153052+00:00", "level": "INFO"}
{"message": "Application shutdown complete.", "asctime": "2025-04-15 08:56:04,153", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "shutdown", "lineno": 76, "timestamp": "2025-04-15T08:56:04.153252+00:00", "level": "INFO"}
{"message": "Finished server process [296]", "asctime": "2025-04-15 08:56:04,153", "levelname": "INFO", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "_serve", "lineno": 93, "color_message": "Finished server process [\u001b[36m%d\u001b[0m]", "timestamp": "2025-04-15T08:56:04.153381+00:00", "level": "INFO"}
{"message": "Stopping reloader process [294]", "asctime": "2025-04-15 08:56:04,164", "levelname": "INFO", "name": "uvicorn.error", "module": "basereload", "filename": "basereload.py", "funcName": "shutdown", "lineno": 112, "color_message": "Stopping reloader process [\u001b[36m\u001b[1m294\u001b[0m]", "timestamp": "2025-04-15T08:56:04.164515+00:00", "level": "INFO"}

However in process list I have zombie python process for some reason:

⬢ [Docker] ❯ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
app            1  0.0  0.0   2940  1408 ?        Ss   08:54   0:00 tail -f /dev/null
app           47  0.0  0.0   6420  4864 pts/0    Ss+  08:54   0:00 zsh
app          239  0.2  0.0   6740  5116 pts/1    Ss   08:54   0:00 zsh
app          295  0.0  0.0      0     0 pts/0    Z    08:54   0:00 [python] <defunct>
app          412  0.0  0.0   8484  4352 pts/1    R+   08:56   0:00 ps aux

With TERM as stop signal I get same issue. I also tried KILL and QUIT and this did not left zombie processes, but it did not close all uvicorn processes like those:

app          295  0.1  0.0  20736 16728 pts/0    S    08:54   0:00 /opt/venvs/app/bin/python -B -c from multiprocessing.resource_tracker import main;main(4)
app          296 10.5  0.5 558180 184344 pts/0   Sl   08:54   0:06 /opt/venvs/app/bin/python -B -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=5, pipe_handle=7) --multiprocessing-fork

So how can I properly close uvicorn app on supervisor exit?

Some package info:

⬢ [Docker] ❯ poetry show uvicorn           
 name         : uvicorn                         
 version      : 0.34.1                          
 description  : The lightning-fast ASGI server. 

⬢ [Docker] ❯ poetry show supervisor     
 name         : supervisor                                        
 version      : 4.2.5                                             
 description  : A system for controlling process state under UNIX 

⬢ [Docker] ❯ poetry env info

Virtualenv
Python:         3.12.9
Implementation: CPython
Path:           /opt/venvs/app
Executable:     /opt/venvs/app/bin/python
Valid:          True

Base
Platform:   linux
OS:         posix
Python:     3.12.9
Path:       /usr/local
Executable: /usr/local/bin/python3.12

// EDIT

Did some more testing and it seems it is not related to supervisor at all. I've run:

⬢ [Docker] ❯ uvicorn aiapi.main:app --host 0.0.0.0 --port 8080    --log-config src/logging.conf    --reload --reload-dir src    --timeout-graceful-shutdown 0
{"message": "Will watch for changes in these directories: ['/app/src']", "asctime": "2025-04-15 11:37:33,042", "levelname": "INFO", "name": "uvicorn.error", "module": "config", "filename": "config.py", "funcName": "__init__", "lineno": 318, "timestamp": "2025-04-15T11:37:33.042756+00:00", "level": "INFO"}
{"message": "Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)", "asctime": "2025-04-15 11:37:33,042", "levelname": "INFO", "name": "uvicorn.error", "module": "config", "filename": "config.py", "funcName": "bind_socket", "lineno": 524, "color_message": "Uvicorn running on \u001b[1m%s://%s:%d\u001b[0m (Press CTRL+C to quit)", "timestamp": "2025-04-15T11:37:33.042997+00:00", "level": "INFO"}
{"message": "Started reloader process [319] using WatchFiles", "asctime": "2025-04-15 11:37:33,043", "levelname": "INFO", "name": "uvicorn.error", "module": "basereload", "filename": "basereload.py", "funcName": "startup", "lineno": 80, "color_message": "Started reloader process [\u001b[36m\u001b[1m319\u001b[0m] using \u001b[36m\u001b[1mWatchFiles\u001b[0m", "timestamp": "2025-04-15T11:37:33.043139+00:00", "level": "INFO"}



{"message": "Started server process [321]", "asctime": "2025-04-15 11:37:38,348", "levelname": "INFO", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "_serve", "lineno": 83, "color_message": "Started server process [\u001b[36m%d\u001b[0m]", "timestamp": "2025-04-15T11:37:38.348986+00:00", "level": "INFO"}
{"message": "Waiting for application startup.", "asctime": "2025-04-15 11:37:38,349", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "startup", "lineno": 48, "timestamp": "2025-04-15T11:37:38.349273+00:00", "level": "INFO"}
{"message": "Application startup complete.", "asctime": "2025-04-15 11:37:38,349", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "startup", "lineno": 62, "timestamp": "2025-04-15T11:37:38.349451+00:00", "level": "INFO"}
{"message": "3 changes detected", "asctime": "2025-04-15 11:37:39,833", "levelname": "INFO", "name": "watchfiles.main", "module": "main", "filename": "main.py", "funcName": "_log_changes", "lineno": 308, "timestamp": "2025-04-15T11:37:39.833735+00:00", "level": "INFO"}

Process list looked like this:

⬢ [Docker] ❯ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
app            1  0.0  0.0   2940  1408 ?        Ss   11:37   0:00 tail -f /dev/null
app           46  0.4  0.0   6448  4732 pts/0    Ss   11:37   0:00 zsh
app          265  0.6  0.0   6444  4992 pts/1    Ss   11:37   0:00 zsh
app          319  6.7  0.1 108600 35824 pts/1    Sl+  11:37   0:00 /opt/venvs/app/bin/python /opt/venvs/app/bin/uvicorn aiapi.main:app --host 0.0.0.0 --port 8080 --log-config src/logging.conf --reload --reload-dir src --timeout-graceful-shutdown 0
app          320  0.8  0.0  20764 16884 pts/1    S+   11:37   0:00 /opt/venvs/app/bin/python -B -c from multiprocessing.resource_tracker import main;main(4)
app          321 92.2  0.5 558168 184104 pts/1   Sl+  11:37   0:06 /opt/venvs/app/bin/python -B -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=5, pipe_handle=7) --multiprocessing-fork
app          331  0.0  0.0   8484  4352 pts/0    R+   11:37   0:00 ps aux

Now CTRL+C

^C{"message": "Shutting down", "asctime": "2025-04-15 11:37:47,175", "levelname": "INFO", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "shutdown", "lineno": 263, "timestamp": "2025-04-15T11:37:47.175502+00:00", "level": "INFO"}
{"message": "Cancel 0 running task(s), timeout graceful shutdown exceeded", "asctime": "2025-04-15 11:37:47,276", "levelname": "ERROR", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "shutdown", "lineno": 283, "timestamp": "2025-04-15T11:37:47.276217+00:00", "level": "ERROR"}
{"message": "Waiting for application shutdown.", "asctime": "2025-04-15 11:37:47,276", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "shutdown", "lineno": 67, "timestamp": "2025-04-15T11:37:47.276503+00:00", "level": "INFO"}
{"message": "Application shutdown complete.", "asctime": "2025-04-15 11:37:47,276", "levelname": "INFO", "name": "uvicorn.error", "module": "on", "filename": "on.py", "funcName": "shutdown", "lineno": 76, "timestamp": "2025-04-15T11:37:47.276726+00:00", "level": "INFO"}
{"message": "Finished server process [321]", "asctime": "2025-04-15 11:37:47,276", "levelname": "INFO", "name": "uvicorn.error", "module": "server", "filename": "server.py", "funcName": "_serve", "lineno": 93, "color_message": "Finished server process [\u001b[36m%d\u001b[0m]", "timestamp": "2025-04-15T11:37:47.276896+00:00", "level": "INFO"}
{"message": "Stopping reloader process [319]", "asctime": "2025-04-15 11:37:47,287", "levelname": "INFO", "name": "uvicorn.error", "module": "basereload", "filename": "basereload.py", "funcName": "shutdown", "lineno": 112, "color_message": "Stopping reloader process [\u001b[36m\u001b[1m319\u001b[0m]", "timestamp": "2025-04-15T11:37:47.287156+00:00", "level": "INFO"}

And there is a zombie process :/

⬢ [Docker] ❯ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
app            1  0.0  0.0   2940  1408 ?        Ss   11:37   0:00 tail -f /dev/null
app           46  0.3  0.0   6448  4732 pts/0    Ss   11:37   0:00 zsh
app          265  0.4  0.0   6444  4980 pts/1    Ss+  11:37   0:00 zsh
app          320  0.4  0.0      0     0 pts/1    Z    11:37   0:00 [python] <defunct>
app          382  0.0  0.0   8484  4224 pts/0    R+   11:37   0:00 ps aux

Any way to properly close uvicorn?