Give apps RAM weighting

[lukechilds]

The app store is beginning to grow large enough that even 4GB RAM devices can no longer run all apps simultaneously without exceeding memory. We have a generous 4GB swapfile on Umbrel OS so this isn't causing crashes, but it will cause heavily degraded performance.

We should add some sort of RAM weighting to each app in the app manifest. This should be a rough estimate of how much memory an app uses during idle +~10% for some headroom. The total RAM weight for all installed apps should not exceed the total available RAM on the system. If a user tries to install an app that will push them over the limit they'll receive an error stating their device does not have enough RAM to install this app and they'll need to uninstall other apps to make room.

The RAM weight just needs to be a rough estimate of idle usage to prevent devices running loads of apps and processes being killed or everything being pushed into swap. It doesn't need to be accurate, and it doesn't matter if they regularly exceed those limits. e.g if an app exceeds the limit when being actively used but not at idle, that's fine. Normally only a small amount of apps are bring actively used at a time so if one or two are over the threshold everything should still be within RAM.

The RAM weighting is a temporary solution that allows us to quickly prevent users running into memory issues. Long term we should have a more sophisticated solution by monitoring the memory usage of individual apps and prompting the user to upgrade RAM or uninstall the problematic app if they are frequently running out of memory (ideally before that point).

[AaronDewes]

Related is this: getumbrel/umbrel-os#189, which would make it easier to limit app memory.

[AaronDewes]

The app store should probably be locked until after sync or, when we can detect that, until electrs is finished because IBD uses quite a lot of RAM.

[lukechilds]

Related is this: getumbrel/umbrel-os#189, which would make it easier to limit app memory.

Ahh, is this what's preventing memory usage showing up in docker stats on Umbrel OS?

The app store should probably be locked until after sync or, when we can detect that, until electrs is finished because IBD uses quite a lot of RAM.

I think maybe we should take the opposite approach and limit the amount of RAM electrs/Core use during IBD so the app store works reliably. Many apps are functional instantly after boot, such as all LND backed apps, so it's not great UX to prevent users from using them until IBD is complete (@mayankchhabra thoughts?).

Giving Core huge amounts of RAM made sense when it was pretty much the only resource intensive thing running. It doesn't make as much sense now that it's just one service out of many others on Umbrel. Also, while I haven't tested this, I'm skeptical of how much difference the increased dbcache makes on Pi hardware since the main bottleneck is CPU. So we might be making a huge memory tradeoff for only a very small perf increase.

[AaronDewes]

Ahh, is this what's preventing memory usage showing up in docker stats on Umbrel OS?

Probably.

I think maybe we should take the opposite approach and limit the amount of RAM electrs/Core use during IBD so the app store works reliably. Many apps are functional instantly after boot, such as all LND backed apps.
Giving Core huge amounts of RAM made sense when it was pretty much the only resource intensive thing running. It doesn't make as much sense now that it's just one service is many others. Also, while I haven't tested this, I'm skeptical of how much difference the increased dbcache makes on Pi hardware since the main bottleneck is CPU. So we might be making a huge memory tradeoff for only a very small perf increase

👍

[lestephane]

What commands do you concretely need people to run to gather the intel you need? If you had a statistics gathering mechanism similar to the troubleshooting aid, that can (with optin from the user) send statistics back to getumbrel without non-tech users fiddling with SSH, we could get much faster there.

[lukechilds]

@lestephane great idea.

We're actually planning on rolling out a preview release to a few opt-in users over the next few days which could include a memory monitoring script.

We just need to loop over each installed app, list all the pids from each container, look up memory usage of those processes on the host, and save the results with the date. Then loop that maybe once a minute or something. I haven't written anything up yet but it's high on my list of priorities.

[lestephane]

Count me in to try the memory monitoring script. I'm worried about adding any new app beyond the ones I have until I understand the impact of each. Maybe we'll end up with a memory usage stacked line chart (broken down by app) in the end? Here's to me hoping...

[DownRangeDevOps]

Is this issue to specifically add memory/cpu limits to apps, or to also add them to the main compose file? I'd like to clean up the compose file and add mem/cpu limits unless someone else is already doing it. As it stands, bitcoind and lnd can eat up all the CPU/Mem (especially during initial sync) and crash the docker daemon.

[IMPranshu]

The app store is beginning to grow large enough that even 4GB RAM devices can no longer run all apps simultaneously without exceeding memory. We have a generous 4GB swapfile on Umbrel OS so this isn't causing crashes, but it will cause heavily degraded performance.

We should add some sort of RAM weighting to each app in the app manifest. This should be a rough estimate of how much memory an app uses during idle +~10% for some headroom. The total RAM weight for all installed apps should not exceed the total available RAM on the system. If a user tries to install an app that will push them over the limit they'll receive an error stating their device does not have enough RAM to install this app and they'll need to uninstall other apps to make room.

The RAM weight just needs to be a rough estimate of idle usage to prevent devices running loads of apps and processes being killed or everything being pushed into swap. It doesn't need to be accurate, and it doesn't matter if they regularly exceed those limits. e.g if an app exceeds the limit when being actively used but not at idle, that's fine. Normally only a small amount of apps are bring actively used at a time so if one or two are over the threshold everything should still be within RAM.

Can we have a terminate option to terminate any app when not in use? As most of the apps store data persistently we can just terminate/kill apps from the background at any given point to free the memory. Essentially what it will do is to stop the running containers and free up RAM altogether and when the user opens a buffer screen may be shown to let users know that the app is restarting.

For example -
I am running a BTC Full node. Let's say I terminate the BTC Node app so my container stops but the data is stored, now when I start the app it will have a screen to let me know that the container is getting started and then starts resyncing the node as it usually does when umbrel shutdown and/or restarts. Thoughts

The RAM weighting is a temporary solution that allows us to quickly prevent users running into memory issues. Long term we should have a more sophisticated solution by monitoring the memory usage of individual apps and prompting the user to upgrade RAM or uninstall the problematic app if they are frequently running out of memory (ideally before that point).

In the future, we can intelligently kill apps based on last usage and/or frequency of usage to automatically free up memory just like how iOS does, so that the users won't have to go on and kill apps one by one as we do in Android devices.