Scaling ElastAlert2 to run 1000s of rules

[rpotnuru]

Hello,

Want to see if there is a clean solution of scaling the application to run thousands of rules. We are running more than 500 rules and this is making the container overwhelming and scheduler is failing to finish the jobs with in the default run cycle.

Thanks, Ravi Potnuru.

[jertel]

Have you checked the load metrics on the Elastic cluster?

[rpotnuru]

Our OpenSearch Clusters are huge and they are able to handle the load, but we are running into some challenges with APScheduler.
I am thinking one pod may be able to handle handful of rules but not 1000s of rules. We have implemented some workarounds to allocate more pods based on the rules count, but that is not optimal and ending up having too many pods and wasting system resources. Hence, I wanted to start a discussion here, to see if someone ran into similar situation and how they solved it.

[jertel]

I've not run several thousand rules but I have run hundreds and low 1k's.

Is the physical node that's running the ElastAlert2 pod getting maxed out on CPU? If not, how many threads/executors are you using for the scheduler? What is the run_every set to?

[rpotnuru]

Hey @jertel , we have ~5K rules, and definitely one pod is not able to handle them. Sorry, was busy and couldn't get back to you on this earlier.
Am using ThreadPoolExecutor with 'max_threads' set to 20
Run ever we increased from 60 to 120 to see if that helps.