changedetection.io getting OOM-killed

[ginkel]

Describe the bug
Starting around a one or two weeks ago, my changedetection.io instance started periodically falling victim to Docker's OOM killer (every one to two days). Before, I had configured its Docker container with a memory limit of 512 MB (which worked fine since installing it > 1 year ago), now I have increased it to 1 GB, but after a while the Docker container is still getting killed.

As initial memory consumption is ~ 300 MB, this looks somewhat like a kind of memory leak or spike.

Around the time the error happend I see some scraping activity in the log, but nothing out of the ordinary caught my eye.

Any ideas how to further analyze this issue?

Version
v0.49.4

How did you install?
Docker on Ubuntu 24.04

[dgtlmoon]

hmm some sites cause libxml (used for processing the markup) to grow.. this is sort of not directly a problem with changedetection.io , i would suggest trying to load sites slowly into a new instance and find out which one "blows it out"

[dgtlmoon]

There was this PR a long time ago, i was thinking of making it an on/off option https://github.com/dgtlmoon/changedetection.io/pull/2659/files

[ginkel]

Thanks! I have one site being monitored that was quite new. Once I switched that to Playwright the OOM issues stopped.

[dgtlmoon]

Thanks! I have one site being monitored that was quite new. Once I switched that to Playwright the OOM issues stopped.

I dont understand, so when you used chrome type fetcher (playwright) it didnt use much memory, but when you used the plaintext/fast/requests one it did?

[ginkel]

Exactly. I can switch back to "Basic fast Plaintext/HTTP Client", just to make sure

[ginkel]

Just got an OOM with the "Basic fast Plaintext/HTTP Client":

Unfortunately, I can't share the monitored site for privacy reasons :-( For now I will switch it back to Playwright.

[Tarow]

I have a similar issue. I am only monitoring a single website at this point (https://de.amazfit.com/products/amazfit-active-2?variant=46424581308586) via Playwright (sockpuppetbrowser).

Monitoring the pod, i can see the there is definitely a leak:

Every time the the job runs memory consumption increases and it's not being freed again.

[dgtlmoon]

@Tarow thanks, interesting, yeah as I suspected there are quite few filters, if you remove all the filters and run it again, can you recreate the memory problem? (or pause all the watches, make a new watch with NO filters and check the memory use over time)

[Tarow]

I removed all filters and set the interval to 5 minutes to get frequent runs.
Now this is what it looks like:

First 3 runs the memory was only rising a little bit (but still looked like a small leak): 253MB -> 267MB -> 284MB, and then a bigger jump to 446MB. I will let it keep running now and see how far/quickly it rises.

This is what the new monitor looks like: https://changedetection.io/share/f-E0BAHXB5ka
I paused the original one, so that one is not running.

[dgtlmoon]

something about 180-280Mb is about normal, from what I can tell everything above 80Mb is some kind of libxml/parser cache stuff that cant be released by python properly

[dgtlmoon]

The tradeoff with running the filters in their own "fork" is a LOT more CPU usage because libxml has to reparse/recache all the possible xpath/css stuff

[Tarow]

I'm surprised that clearing all filters doesn't seem to have any effect tho.
Even with the new config and no filters. the RAM usage keeps rising:

But i guess if it's actually related to libxml parser then i'll have to wait for an upstream fix from their side. Thanks for your efforts!

[xconverge]

Piling on here I am also experiencing a leak that I haven't pinned down, my changedetection container is up to 1.4G ram usage currently. I have 17 watches, 13 are plain http and 4 are using sockpuppetbrowser

[xconverge]

0.49.1 looks good (low usage and stays low), trying 0.49.4 now and it seems to exhibit the memory creep issue

@dgtlmoon I have some suspicions around #2999 I am not sure this would corroborate or relate to the information above of switching from "Basic fast Plaintext/HTTP Client" to playrwright though, because this PR only impacts playwright watches I think?

I don't want to start a wild goose chase but I would STRONGLY look through this changeset

For now I am going to run 0.49.3 and 0.49.4 for a few days and see how it looks, but SO FAR it looks like 0.49.3 is good and 0.49.4 has memory issues for me

[xconverge]

@ginkel (if you are interested and have time) can you test 0.49.3 and see if you still experience the problem, for me I have pinned it down to running 0.49.3 is good and 0.49.4 has memory issues on my system

[ginkel]

I have downgraded to 0.49.3 and will monitor how it behaves...

[ginkel]

Deployed 0.49.3 yesterday at 19:30 the result is quite impressive - memory no longer drastically increasing:

[xconverge]

@ginkel is it still under control? do you have any playwright watches that are quite tall, if you click "History" or "Preview" for the watch, then click the Screenshot tab, is there warning text at the top that says it is X pixels tall?

I have this issue primarily with a watch that is 77000px tall. The issues I am seeing and the main change (related to how screenshots are done in playwright between 0.49.3 and 0.49.4) seem to only impact this scenario, but your other thread about switching AWAY FROM playwright resolving it wouldn't make sense for this...

[dgtlmoon]

by the way all- "memory usage" is not so simple!!! it can be different kinds of memory usage

• resident memory
• shared memory
• cached memory
• virtual memory

please try to work out which kind your tools are reporting!!

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                  
  68106 root      20   0   19.4g 414676  17496 S   2.3   0.3 196:30.24 dockerd                                                                                                                                                                  
 123903 root      20   0 4710260 267508  64116 S   0.0   0.2  26:20.78 containerd                                                                                                                                                               
2649153 root      20   0  859260 213372  19900 S   0.0   0.2   0:17.13 python           

here it is 859mb "virtual" memory but 213Mb Resident (actual memory) !

[xconverge]

My graphs and logging are matching what docker stats is reporting. I believe that this is simply the memory being used by the container. I believe this includes resident and cached at least. I do not think virtual memory is in play with my analysis and perceived ram usage.

[dgtlmoon]

can anyone confirm if the memory still grows even when NO sites change? that would be interesting

[ginkel]

For my monitored sites, the last change was 1 week ago. So I am seeing growing memory consumption without any site changes.

[dgtlmoon]

Can you guys try the latest :dev container? it's set to use python 3.11 from 3.10, probably it doesnt help much but it shouldnt be worse :)

[xconverge]

I see the same behavior with :dev, see my thread above but give me another day or two and I am sure I am honing in on something that might give us a better clue/idea

[dgtlmoon]

It has to be related to the sites you're following, its the only variable, we didnt see this on the paid subscriptions (of which there are tens of thousands of web pages being followed), in both older and newer versions of the container

[dgtlmoon]

Also its not clear what kind of memory is the graph showing you, if it includes cached memory, shared memory etc etc

[dgtlmoon]

I can see one instance where even flushing the caches only saved a few megabytes of ram hmm

[dgtlmoon]

try to reproduce the issue with only one URL in your config that has a very very long page

[xconverge]

I can.

[xconverge]

v0.49.13 looks good, thanks @dgtlmoon