Possible double caching -- how to investigate?

[richardm1]

I've got an app that reads ~6GB files through a mergerfs mount. After a read the system's page cache will have grown by ~12GB. I've confirmed cache.files=off.

I can cp one of these files to /dev/null through the same mergerfs mount and cache will have grown by only ~6GB.

If it makes a difference I strongly suspect the app is doing memory-mapped reads.

Is there a kernel tracing knob I can twist to gain visibility into what might be happening here? I think it'd be helpful to see what fs calls are being made. The app was created with Avalonia framework and accessing remote filesystems and through symlinks and such has always been a little screwy and problematic.

[trapexit]

As explained in the docs double caching is an unfortunately but natural consequence of being a FUSE server and interacting with local filesystems. mergerfs is a filesystem that has a page cache as does the underlying filesystem. That's why dropcacheonclose exists.

The nocache project has cachestats app which can show you some details and if you want to see syscalls you can use strace.

I could add additional features to minimize cache bloat but need to understand the particular situation / set of options.

[richardm1]

The docs imply mmap read caching doesn't work in certain scenarios thus the need to enable cache.files to get any caching at all. I didn't see anything about double caching (or I keep missing it).

I did just set dropcacheonclose and it seems to be an improvement.

[trapexit]

It isn't yelled from the rooftops but is mentioned in a couple places:

• https://trapexit.github.io/mergerfs/latest/faq/technical_behavior_and_limitations/#why-use-fuse-why-not-a-kernel-based-solution
  • "Double caching when using page caching"
• https://trapexit.github.io/mergerfs/latest/config/options/#mount-options
  • dropcacheonclose=BOOL: When a file is requested to be closed call posix_fadvise on it first to instruct the kernel that we no longer need the data and it can drop its cache. Recommended when cache.files=partial|full|auto-full|per-process to limit double caching. (default: false)
• https://trapexit.github.io/mergerfs/latest/known_issues_bugs/#cached-memory-appears-greater-than-it-should-be
• https://trapexit.github.io/mergerfs/latest/config/cache/
  • "Generally, enabling the page cache actually harms performance1. In part because it can lead to buffer bloat due to the kernel caching both the underlying filesystem's file content as well as the file through mergerfs."

[richardm1]

Got it. So I've landed on dropcacheonclose=true along with cache.files=full. Any other combination will either double cache and hold onto both copies or else drop the cached pages entirely -- either 1-2s after the read completes or else later after the file is closed.

"true + full" double caches then drops one copy right after the read finishes. The other copy is held indefinitely until the pages age-out. It's the least problematic of the four combinations. I see FUSE caching behavior is getting a lot of attention from both Linux plumbers and from the Android side. I reckon we'll have a fix at some point.

[trapexit]

It's not just FUSE. It's caching in general. FUSE is just another filesystem and any FUSE filesystem interacting with local filesystem, like mergerfs, is just like having two sets of filesystems with the same data. There isn't much that could be done without a massive change to what FUSE is.

There isn't any practical way to disable page caching broadly. O_DIRECT has awkward requirements/restrictions and has to be on the caller side, advise(DONTNEED) is also on caller side, RWF_UNCACHED was only just introduced and similarly requires the software to be involved, and DAX doesn't fit this situation. I can only control what mergerfs does and that means advise(DONTNEED) or RWF_UNCACHED reads/writes.

Can you share what exactly you did to check this out? cache.files=off specific disables page caching on the FUSE side and dropcacheonclose would impact what mergerfs is acting on.

[trapexit]

The only way for mergerfs to manage the cache of those things accessing mergerfs would be to create some special cache killing daemon that goes and watches for accessed files and DONTNEEDs them after close. Which really wouldn't be mergerfs specific.

[trapexit]

I just used the nocache project's cachestats and confirmed that when cache.files=off mergerfs is not cached. With dropcacheonclose=true it clears out the cache on file close. cachestats shows after reading a file through mergerfs with those settings that they have nothing kept around.

[trapexit]

BTW... in the next release I'm adding passthrough mode which effectively ties the page cache of the two together. I just tested with cachestats and interestingly if you try to cachedel the file through mergerfs it simply doesn't do anything. If the underlying filesystem file is the target both clear. Meaning with dropcacheonclose=true with passthrough enabled leads to a singular page cache for the pair of filesystems and once closed it is cleared.

[richardm1]

Can you share what exactly you did to check this out?

I'm using ComfyUI (Stable Diffusion). Safetensor checkpoints are around 6.5 to 7GB.

[rm@cachyos comfy]$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3
[rm@cachyos comfy]$ cat /proc/meminfo | grep -i cached
Cached:          4975428 kB

Fire up an inference job which loads the checkpoint file and a few other odds and ends. Then...

[rm@cachyos comfy]$ cat /proc/meminfo | grep -i cached
Cached:         21993016 kB

Here's a visual. This is cache.files=off and dropcacheonclose=false. The big dip is 3 > drop_caches then the ramp-up is nearly 2x the actual data read:

Here's cache.files=full and dropcacheonclose=true:

[rm@cachyos /]$ echo 3 | sudo tee /proc/sys/vm/drop_caches
3
[rm@cachyos /]$ cat /proc/meminfo | grep -i cached
Cached:          4849652 kB

Run the image generation then re-check cache size:

[rm@cachyos /]$ cat /proc/meminfo | grep -i cached
Cached:         14650912 kB    # this was  21993016 kB last time

The visual. As before, the cache ramps roughly double what's expected. Then ~40% is discarded immediately after the reads are complete:

Switch to a different checkpoint (also ~6.5GB). Note double read then immediate drop by half:

Also, I've tried a hack to disable mmap reads and I can confirm it completely solves the double caching when used with cache.files=off and dropcacheonclose=false.

[trapexit]

This test, while interesting, is not something that I can easily reproduce.

Regardless, mmap doesn't work when cache.files=off in the version of mergerfs you have. So the software must be doing something different between these setups. And if it did work... it would be just automagically enabling page caching and should behave the same as cache.files=full.

[richardm1]

I can barely spell strace so I'm not entirely sure what I'm looking at. This is from a "pathological" config where mmaps are allegedly being performed.

[pid 46018] openat(AT_FDCWD, "/mnt/storage-ssd/SM/Data/Models/StableDiffusion/realismIllustriousBy_v40FP16.safetensors", O_RDONLY|O_CLOEXEC) = 53
[pid 46018] statx(53, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=6938040880, ...}) = 0
[pid 46018] mmap(NULL, 6938040880, PROT_READ, MAP_PRIVATE, 53, 0) = 0x72fb4e600000
[pid 45987] <... futex resumed>)        = -1 ETIMEDOUT (Connection timed out)
[pid 45987] futex(0x72fe675bb540, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 45987] futex(0x72fe675bb530, FUTEX_WAIT_BITSET_PRIVATE, 38, {tv_sec=26204, tv_nsec=111814892}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 46018] rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
[pid 46018] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 46018] getrandom("\x0b\x2c\x7a\x45\x6d\x5a\x9d\xce\x58\xfb\xf3\x7e\xb6\x2e\x6a\x50\xdb\xba\x68\xd2\x82\x5a\x20\xf6\x19\x66\x73\xaf\xe3\xb8\xdf\x93", 32, 0) = 32
[pid 46018] mprotect(0x72fcec021000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec022000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec023000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec024000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec025000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec026000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec027000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec028000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcec029000, 4096, PROT_READ|PROT_WRITE) = 0
...
pid 46018] mprotect(0x72fcede19000, 1236992, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcedf47000, 7372800, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcee64f000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcee717000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcee7df000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcee8a7000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcee96f000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fceea37000, 6553600, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcef077000, 3276800, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcef397000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcef45f000, 2621440, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcef6df000, 2621440, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcef95f000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x72fcf0506000
[pid 46018] mprotect(0x72fcefa27000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcefaef000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcefbb7000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fcefc7f000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x72fce4000000
[pid 46018] munmap(0x72fce8000000, 67108864) = 0
[pid 46018] mprotect(0x72fce4000000, 6688768, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce4661000, 3145728, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce4961000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce4a29000, 2621440, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce4ca9000, 2621440, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce4f29000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce4ff1000, 819200, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce50b9000, 7372800, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce57c1000, 1638400, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce5951000, 7372800, PROT_READ|PROT_WRITE) = 0
[pid 46018] mprotect(0x72fce6059000, 819200, PROT_READ|PROT_WRITE) = 0

With the hack applied strace output is downright confusing. The mmap(),mprotect(), and munmap() don't appear until much later in the capture; after the disk reads are basically done. I don't think I'm using this tool correctly. It's probably not even the correct tool.

Anyway, I have a fix for the time being. Thanks for a great product -- I truly think it's the very best at what it does.

[trapexit]

strace just spits out what system calls the app is making to the Linux kernel.

mmap(NULL, 6938040880, PROT_READ, MAP_PRIVATE, 53, 0) = 0x72fb4e600000

So this is the call of interest. And this explains why it works with cache.files=off. It's using MAP_PRIVATE which wasn't what I expected but makes sense if all they are doing is reading the data. FUSE can't work with mmap when using a different flag (MAP_SHARED).

This gives me something to work with but still unclear to me why the cached amount would be more. I might need to reach out to the kernel developers but will have to build some simple examples to show it happen. Unfortunately, the app could be doing other things that might impact it.