fapolicyd consuming excess CPU

[davidstuffle]

Running on RHEL8 AWS i3en.24xlarge instances, fapolicyd takes the CPU load from around 10 up to over 200, or until we stop fapolicyd. Our suspicion is that we have a xfs file system mounted to S3 that manages the "smartstore" cache for Splunk indexers and it is extremely busy loading and evicting cache which fapolicyd can't keep up with. To verify this, we were looking for a way to exclude this particular file system. watch_fs doesn't seem to allow this as xfs is used for several of the file systems. Is there a way to exclude a specific file system? fapolicyd caches and queue settings have already been increased.

[stevegrubb]

There's not a good answer for this one without some development work. You could not watch that type of file system. Or put the indexer's data on ext4 and not watch that. If it could be worked around in rules, it would still affect the internal cache. Once in a while, you run across a workload that it's not suited for - like software build hosts. We could make a configuration option to drop certain mount points but it would take a while to be released and shipped. It would be the best solution in this case since that would avoid the cache.

[davidstuffle]

Thanks Steve. If we've increased the cache, verified the trust db, disabled integrity checking, run in permissive mode, and still get CPU through the roof, is it safe to say it just doesn't play well Splunk indexers on smartstore and we should just leave it off?

[stevegrubb]

If it is showing a lot of cache ejections and most of the cache is the indexer mount point, the only way to not watch it is for that mount point to be a file system that you can remove from the watched set. If that's not doable, then the only option is not to use it. In the mean time, we'll put it on the road map to allow exclusion by mount point path.

[davidstuffle]

I appreciate that Steve. I'll keep an eye out for that feature. Thanks for the help.

[radosroka]

@davidstuffle what about blacklisting mountpoint?

I have worked on implementation of such a feature in #329. It's not finished yet.
What do you think?

[davidstuffle]

Yes...that looks like it would solve our problem!