Memory leak in a webserver? #135

abey79 · 2022-05-23T10:39:53Z

abey79
May 23, 2022

In order to address the relatively long startup time of lib camera-still, I made a quick-and-dirty FastAPI-based server looking roughly like this:

app = FastAPI()

# Initialize the camera
picam2 = Picamera2()
still_config = picam2.still_configuration(controls={"ExposureValue": 0})
picam2.configure(still_config)
picam2.start()

@app.get("/img/")
def read_item(ev: int = 0):
    picam2.set_controls({"ExposureValue": ev})
    array = picam2.capture_array()
    image = PIL.Image.fromarray(array)
    data = io.BytesIO()
    image.save(data, format="JPEG")
    data.seek(0)
    return Response(content=data.read(), media_type="image/jpg")

It works great, but it leaks memory. On my RPi3 with 1GB of RAM, I can take maybe ~10 images before the server stalls due to lack of available memory (I tracked the evolution of this with top).

Is there something obvious I'm missing that I should do to avoid leaking?

Answered by abey79

May 25, 2022

It looks like gc.collect() forces memory release, and it appears that PIL is the culprit. Here is a simplified demo script, without FastAPI.

import gc
import io

import numpy as np
import psutil
import PIL
from picamera2 import Picamera2

picam2 = Picamera2()
still_config = picam2.still_configuration(controls={"ExposureValue": 0})
picam2.configure(still_config)
picam2.start()


def get_pic() -> bytes:
    array = picam2.capture_array()
    image = PIL.Image.fromarray(array)
    data = io.BytesIO()
    image.save(data, format="JPEG")
    data.seek(0)
    buffer = data.read()
    return buffer


def get_pic2() -> np.ndarray:
    array = picam2.capture_array()
    return array


print("With …

View full answer

davidplowman · 2022-05-24T16:20:04Z

davidplowman
May 24, 2022
Maintainer

I can't see anything obvious here. It's leaking user space memory (rather than CMA, for example)? Sometimes with Python I've found I have to explicitly delete things (del xyz) to make them go away. It never seems like you should have to do that but I have known it to help.

0 replies

abey79 · 2022-05-24T18:01:15Z

abey79
May 24, 2022
Author

I've been investigating a bit further with tracemalloc and an updated version of my server script:

import io
import tracemalloc

import PIL
from fastapi import FastAPI, Response
from picamera2 import Picamera2

app = FastAPI()

picam2 = Picamera2()
still_config = picam2.still_configuration(controls={"ExposureValue": 0})
picam2.configure(still_config)
picam2.start()

tracemalloc.start()
initial_snapshot = tracemalloc.take_snapshot()


def get_pic(ev: int) -> bytes:
    picam2.set_controls({"ExposureValue": ev})
    array = picam2.capture_array()
    image = PIL.Image.fromarray(array)
    data = io.BytesIO()
    image.save(data, format="JPEG")
    data.seek(0)
    buffer = data.read()
    return buffer


@app.get("/img/")
def read_item(ev: int = 0):
    return Response(content=get_pic(ev), media_type="image/jpg")


@app.get("/trace/")
def trace():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.compare_to(initial_snapshot, 'lineno')
    print("[ Top 10 differences ]")
    for stat in top_stats[:10]:
        print(stat)

I'm making successive requests to the /img/ and /trace/ endpoints to monitor what's going on.

What I observe is that picamera2/request.py:130 leaks about 35MB per capture, but that memory gets somehow magically released just after a call to /trace/ (side effect of tracemalloc???). So it goes:

/img/
/trace/ -> ~35MB
/img/
/trace/ -> ~35MB (no accumulation!!)
/img/
/img/
/trace/ -> ~70MB
/img/
/img/
/img/
/trace/ -> ~105MB

Monitoring top in the process, I can indeed see that /trace/ releases some memory.

Here is a full transcript with the tracemalloc print displayed just after the related INFO: GET request log line:

$ uvicorn --host 192.168.2.130 campi_simple:app --reload
INFO:     Will watch for changes in these directories: ['/home/pi/src/campi']
INFO:     Uvicorn running on http://192.168.2.130:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [1004] using watchgod
[0:01:43.512586119] [1006]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3544-22656360
[0:01:43.556174306] [1010]  WARN CameraSensorProperties camera_sensor_properties.cpp:163 No static properties available for 'imx477'
[0:01:43.556325140] [1010]  WARN CameraSensorProperties camera_sensor_properties.cpp:165 Please consider updating the camera sensor properties database
[0:01:43.556425974] [1010] ERROR CameraSensor camera_sensor.cpp:591 'imx477 10-001a': Camera sensor does not support test pattern modes.
[0:01:43.601395267] [1010]  INFO RPI raspberrypi.cpp:1356 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
[0:01:43.605863534] [1006]  INFO Camera camera.cpp:1029 configuring streams: (0) 4032x3040-BGR888
[0:01:43.606724271] [1010]  INFO RPI raspberrypi.cpp:760 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-pBCC
INFO:     Started server process [1006]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
[ Top 10 differences ]
<frozen importlib._bootstrap_external>:587: size=107 KiB (+107 KiB), count=1232 (+1232), average=89 B
/usr/lib/python3.9/abc.py:102: size=29.5 KiB (+29.5 KiB), count=301 (+301), average=100 B
/usr/lib/python3.9/abc.py:85: size=23.5 KiB (+23.5 KiB), count=107 (+107), average=225 B
/usr/lib/python3.9/dataclasses.py:400: size=8927 B (+8927 B), count=92 (+92), average=97 B
/usr/lib/python3.9/typing.py:698: size=8640 B (+8640 B), count=120 (+120), average=72 B
/usr/lib/python3.9/abc.py:86: size=6560 B (+6560 B), count=43 (+43), average=153 B
/home/pi/src/campi/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py:85: size=4632 B (+4632 B), count=1 (+1), average=4632 B
/usr/lib/python3.9/asyncio/runners.py:44: size=3904 B (+3904 B), count=16 (+16), average=244 B
/usr/lib/python3.9/typing.py:726: size=3232 B (+3232 B), count=60 (+60), average=54 B
/usr/lib/python3.9/typing.py:519: size=3004 B (+3004 B), count=25 (+25), average=120 B
INFO:     192.168.2.24:50922 - "GET /trace/ HTTP/1.1" 200 OK
INFO:     192.168.2.24:50924 - "GET /img/ HTTP/1.1" 200 OK
[ Top 10 differences ]
/home/pi/src/campi/venv/lib/python3.9/site-packages/picamera2/request.py:130: size=35.1 MiB (+35.1 MiB), count=3 (+3), average=11.7 MiB
/home/pi/src/campi/venv/lib/python3.9/site-packages/PIL/ImageFile.py:519: size=625 KiB (+625 KiB), count=1 (+1), average=625 KiB
./campi_simple.py:26: size=555 KiB (+555 KiB), count=1 (+1), average=555 KiB
<frozen importlib._bootstrap_external>:587: size=486 KiB (+486 KiB), count=5265 (+5265), average=94 B
/usr/lib/python3.9/tracemalloc.py:115: size=45.8 KiB (+45.8 KiB), count=586 (+586), average=80 B
/home/pi/src/campi/venv/lib/python3.9/site-packages/PIL/ImageFile.py:518: size=43.5 KiB (+43.5 KiB), count=2 (+2), average=21.8 KiB
/usr/lib/python3.9/abc.py:85: size=37.0 KiB (+37.0 KiB), count=143 (+143), average=265 B
/usr/lib/python3.9/abc.py:102: size=29.4 KiB (+29.4 KiB), count=299 (+299), average=101 B
/usr/lib/python3.9/tracemalloc.py:193: size=16.6 KiB (+16.6 KiB), count=354 (+354), average=48 B
/usr/lib/python3.9/typing.py:698: size=8640 B (+8640 B), count=120 (+120), average=72 B
INFO:     192.168.2.24:50927 - "GET /trace/ HTTP/1.1" 200 OK
INFO:     192.168.2.24:50929 - "GET /img/ HTTP/1.1" 200 OK
INFO:     192.168.2.24:50932 - "GET /img/ HTTP/1.1" 200 OK
[ Top 10 differences ]
/home/pi/src/campi/venv/lib/python3.9/site-packages/picamera2/request.py:130: size=70.1 MiB (+70.1 MiB), count=5 (+5), average=14.0 MiB
/home/pi/src/campi/venv/lib/python3.9/site-packages/PIL/ImageFile.py:519: size=1274 KiB (+1274 KiB), count=2 (+2), average=637 KiB
./campi_simple.py:26: size=1133 KiB (+1133 KiB), count=2 (+2), average=566 KiB
<frozen importlib._bootstrap_external>:587: size=485 KiB (+485 KiB), count=5258 (+5258), average=95 B
/usr/lib/python3.9/tracemalloc.py:115: size=120 KiB (+120 KiB), count=1538 (+1538), average=80 B
/home/pi/src/campi/venv/lib/python3.9/site-packages/PIL/ImageFile.py:518: size=109 KiB (+109 KiB), count=4 (+4), average=27.2 KiB
/usr/lib/python3.9/abc.py:85: size=37.0 KiB (+37.0 KiB), count=143 (+143), average=265 B
/usr/lib/python3.9/abc.py:102: size=29.4 KiB (+29.4 KiB), count=299 (+299), average=101 B
/usr/lib/python3.9/tracemalloc.py:558: size=19.0 KiB (+19.0 KiB), count=402 (+402), average=48 B
/usr/lib/python3.9/tracemalloc.py:193: size=13.6 KiB (+13.6 KiB), count=290 (+290), average=48 B
INFO:     192.168.2.24:50934 - "GET /trace/ HTTP/1.1" 200 OK
INFO:     192.168.2.24:50936 - "GET /img/ HTTP/1.1" 200 OK
INFO:     192.168.2.24:50939 - "GET /img/ HTTP/1.1" 200 OK
INFO:     192.168.2.24:50941 - "GET /img/ HTTP/1.1" 200 OK
[ Top 10 differences ]
/home/pi/src/campi/venv/lib/python3.9/site-packages/picamera2/request.py:130: size=105 MiB (+105 MiB), count=7 (+7), average=15.0 MiB
/home/pi/src/campi/venv/lib/python3.9/site-packages/PIL/ImageFile.py:519: size=1941 KiB (+1941 KiB), count=3 (+3), average=647 KiB
./campi_simple.py:26: size=1727 KiB (+1727 KiB), count=3 (+3), average=576 KiB
<frozen importlib._bootstrap_external>:587: size=485 KiB (+485 KiB), count=5258 (+5258), average=95 B
/usr/lib/python3.9/tracemalloc.py:115: size=121 KiB (+121 KiB), count=1552 (+1552), average=80 B
/home/pi/src/campi/venv/lib/python3.9/site-packages/PIL/ImageFile.py:518: size=63.3 KiB (+63.3 KiB), count=6 (+6), average=10.5 KiB
/usr/lib/python3.9/abc.py:85: size=37.0 KiB (+37.0 KiB), count=143 (+143), average=265 B
/usr/lib/python3.9/tracemalloc.py:558: size=33.0 KiB (+33.0 KiB), count=700 (+700), average=48 B
/usr/lib/python3.9/abc.py:102: size=29.4 KiB (+29.4 KiB), count=299 (+299), average=101 B
/usr/lib/python3.9/tracemalloc.py:193: size=10.5 KiB (+10.5 KiB), count=225 (+225), average=48 B
INFO:     192.168.2.24:50945 - "GET /trace/ HTTP/1.1" 200 OK

I will try to investigate a bit further later, but at this stage I'm happy to take any pointer you may have.

0 replies

abey79 · 2022-05-25T08:24:40Z

abey79
May 25, 2022
Author

It looks like gc.collect() forces memory release, and it appears that PIL is the culprit. Here is a simplified demo script, without FastAPI.

import gc
import io

import numpy as np
import psutil
import PIL
from picamera2 import Picamera2

picam2 = Picamera2()
still_config = picam2.still_configuration(controls={"ExposureValue": 0})
picam2.configure(still_config)
picam2.start()


def get_pic() -> bytes:
    array = picam2.capture_array()
    image = PIL.Image.fromarray(array)
    data = io.BytesIO()
    image.save(data, format="JPEG")
    data.seek(0)
    buffer = data.read()
    return buffer


def get_pic2() -> np.ndarray:
    array = picam2.capture_array()
    return array


print("With PIL stuff...")
for _ in range(3):
    get_pic()
    print(psutil.virtual_memory())

gc.collect()
print(psutil.virtual_memory())

print("Without PIL stuff...")
for _ in range(3):
    get_pic2()
    print(psutil.virtual_memory())

gc.collect()
print(psutil.virtual_memory())

Output:

[0:53:48.314930198] [1284]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3544-22656360
[0:53:48.352728301] [1288]  WARN CameraSensorProperties camera_sensor_properties.cpp:163 No static properties available for 'imx477'
[0:53:48.352816635] [1288]  WARN CameraSensorProperties camera_sensor_properties.cpp:165 Please consider updating the camera sensor properties database
[0:53:48.352872937] [1288] ERROR CameraSensor camera_sensor.cpp:591 'imx477 10-001a': Camera sensor does not support test pattern modes.
[0:53:48.390184374] [1288]  INFO RPI raspberrypi.cpp:1356 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media3 and ISP device /dev/media0
[0:53:48.393987861] [1284]  INFO Camera camera.cpp:1029 configuring streams: (0) 4032x3040-BGR888
[0:53:48.395342235] [1288]  INFO RPI raspberrypi.cpp:760 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SBGGR12_1X12 - Selected unicam format: 4056x3040-pBCC
With PIL stuff...
svmem(total=953942016, available=558837760, percent=41.4, used=330002432, free=372588544, active=97583104, inactive=253612032, buffers=24543232, cached=226807808, shared=729088, slab=58531840)
svmem(total=953942016, available=471875584, percent=50.5, used=416964608, free=285626368, active=97583104, inactive=340361216, buffers=24559616, cached=226791424, shared=729088, slab=58531840)
svmem(total=953942016, available=384671744, percent=59.7, used=504168448, free=198406144, active=97599488, inactive=427253760, buffers=24559616, cached=226807808, shared=729088, slab=58531840)
svmem(total=953942016, available=606126080, percent=36.5, used=282714112, free=419860480, active=97599488, inactive=203714560, buffers=24559616, cached=226807808, shared=729088, slab=58531840)
Without PIL stuff...
svmem(total=953942016, available=606973952, percent=36.4, used=281866240, free=420708352, active=97599488, inactive=203616256, buffers=24559616, cached=226807808, shared=729088, slab=58531840)
svmem(total=953942016, available=608219136, percent=36.2, used=280621056, free=421953536, active=97599488, inactive=203669504, buffers=24559616, cached=226807808, shared=729088, slab=58531840)
svmem(total=953942016, available=604004352, percent=36.7, used=284835840, free=417738752, active=97599488, inactive=203657216, buffers=24559616, cached=226807808, shared=729088, slab=58531840)
svmem(total=953942016, available=604004352, percent=36.7, used=284835840, free=417738752, active=97599488, inactive=203657216, buffers=24559616, cached=226807808, shared=729088, slab=58531840)

Notice, when using PIL, the memory used percent going up after each capture, and then going back down after gc.collect(). This doesn't seem to happen without using PIL.

0 replies

davidplowman · 2022-05-25T08:39:06Z

davidplowman
May 25, 2022
Maintainer

Thanks for the update. I did try this with a simpler standalone script but didn't see anything bad happen. Maybe it depends a bit what else is going on, I suppose you don't in general have any guarantees about exactly when things get garbage-collected. There are other JPEG encoders too, of course, most obviously OpenCV, I don't know if that might help at all?

0 replies

abey79 · 2022-05-25T08:46:39Z

abey79
May 25, 2022
Author

@davidplowman Well what would help greatly would be to allow file-like objects in picam2.capture_file(), eg:

def get_pic() -> bytes:
    data = io.BytesIO()
    picam2.capture_file(data, format="jpg")
    data.seek(0)
    return data.read()

This would make it super easy to serve image files without writing files (which, in theory, is not very good on flash-based platform like the Raspberry Pi) or using another dependency. For the time being, I reverted to using a temporary file and a FileResponse() for FastAPI. Not ideal, but I'm working on a DIY hack, not some production server. I'd have avoided this entire ordeal if I wasn't so uptight with temp files 😅

3 replies

davidplowman May 25, 2022
Maintainer

Agree, that's a good idea!

davidplowman Jun 1, 2022
Maintainer

This feature has now been implemented on top-of-tree. In the meantime you could either clone it, or do pip3 install git+https://github.com/raspberrypi/picamera2.git to install it.

abey79 Jun 1, 2022
Author

Awesome! I'll be on vacations next couple of week, but I'll sure test it when I'm back!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Memory leak in a webserver? #135

Uh oh!

{{title}}

Uh oh!

Replies: 5 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Memory leak in a webserver? #135

Uh oh!

abey79 May 23, 2022

Replies: 5 comments · 3 replies

Uh oh!

davidplowman May 24, 2022 Maintainer

Uh oh!

abey79 May 24, 2022 Author

Uh oh!

abey79 May 25, 2022 Author

Uh oh!

davidplowman May 25, 2022 Maintainer

Uh oh!

abey79 May 25, 2022 Author

Uh oh!

davidplowman May 25, 2022 Maintainer

Uh oh!

davidplowman Jun 1, 2022 Maintainer

Uh oh!

abey79 Jun 1, 2022 Author

abey79
May 23, 2022

Replies: 5 comments 3 replies

davidplowman
May 24, 2022
Maintainer

abey79
May 24, 2022
Author

abey79
May 25, 2022
Author

davidplowman
May 25, 2022
Maintainer

abey79
May 25, 2022
Author

davidplowman May 25, 2022
Maintainer

davidplowman Jun 1, 2022
Maintainer

abey79 Jun 1, 2022
Author