[BUG] Possible memory leak with Obico `ffmpeg` video streaming

[calexandre]

I've been struggling with an instability issue on my OctoPrint instance, where, from time to time, the system runs out of memory. After some debugging, I managed to isolate the issue to the Obico plugin, specifically the ffmpeg process.

It seems that when I have the streaming enabled (which is the purpose of Obico), the ffmpeg process is automatically spawned and slowly consumes memory until OctoPrint crashes due to out-of-memory issues.

This process takes a few days before memory pressure starts causing instability in the entire system, eventually leading to OctoPrint crashing and failing the print.

I isolated this issue by letting the system run for some time with and without the Obico plugin enabled, which led me to open this issue.

To mitigate this problem, I set up a cron job to kill the ffmpeg process every day.

To Reproduce
Steps to reproduce the behavior:

1. Enable Obico streaming.
2. Monitor the ffmpeg process and check the memory it is consuming.
3. After a couple of days, check the ffmpeg process and notice the sudden increase in memory. Depending on the interval, you may already be experiencing swapping problems due to a lack of available memory.
4. If you don't kill the ffmpeg process, OctoPrint will eventually crash.

Screenshots

Day 0 - Enabled Obico streaming - ffmpeg is consuming almost 4% of memory, which is fine.

Day 1 - After 27 hours, the ffmpeg process is already taking up almost 30% of memory.

Desktop (please complete the following information):

• OS: OctoPi 1.0.0 & OctoPrint 1.9.3 (new camera stack)
• Browser: Chrome
• Obico Version: 2.4.9

Additional context

• I'm a paying customer of the Obico solution, so I have premium streaming enabled.
• I don't know if it is related, but I reported another issue related to OctoPrint reporting lots of errors on the Obico plugin: link to the GitHub issue

[kennethjiang]

Sorry I have been really busy post-Thanksgiving.

This sounds like a pretty bad problem. However I haven't been able to reproduce it.

I notice you are using RTSP stream provided by OctoPrint's new webcam streaming stack. Can you disable it (do the opposite of https://community.octoprint.org/t/got-rtsp-on-octopi-working/54091), restart the Pi, and see if ffmpeg will still have memory leak?

Also do you remember if this problem has always been there, or it started happening at some point? If the latter, do you remember if there was any change to your setup right before this started to happen?

[kennethjiang]

Oh I'm 99% sure the datachannel false alarms in the log file doesn't have to do with this issue. That was an annoying false alarm in Janus, another 3rd-party software that we don't have much control over... :(

[calexandre]

Sorry I have been really busy post-Thanksgiving.

This sounds like a pretty bad problem. However I haven't been able to reproduce it.

I notice you are using RTSP stream provided by OctoPrint's new webcam streaming stack. Can you disable it (do the opposite of https://community.octoprint.org/t/got-rtsp-on-octopi-working/54091), restart the Pi, and see if ffmpeg will still have memory leak?

Also do you remember if this problem has always been there, or it started happening at some point? If the latter, do you remember if there was any change to your setup right before this started to happen?

Sorry for the delay...
I will your suggestion to check if we can narrow it down...

Regarding your other question I cant recall exactly any particular change...this is something i've been tracking for a while (lets say, 6 months max?), because my OctoPrint instance was getting very unstable and I finally started the debug process to find the culprit.

Ever since I created a cronjob watchdog that kills the ffmpeg process every time it reaches 30% memory, the system has been rock solid stable.

[kennethjiang]

It sounds like the issue has been going on for a while now.

I guess you were using the octopi 1.0 SD card image? If so, have you done system upgrade? I'm wondering if it's possible that a system lib was upgraded to a version that causes the memory leak.