Send alert only for first match or how to avoid the realert

[masterplanx]

Hi Team! how are you?
I have 2 rules that monitor the CPU consumption of my kubernetes nodes. A rule that checks if the CPU is above 80% and another rule that checks if the processor is below 40%.
This is working fine I have two alerts set one by slack and the other using http_post, which what it basically does is execute an API that depending on if the value is 80 it scales up and if it is less than 40 it does a downscale, all this is fine.
The problem we are having is that most of the time the nodes are below the min_threshold, therefore the condition is always running. I know elastalert2 doesn't support resolve events, so is there a way to prevent alerts from being re-run? I saw that you can set realert to a high value to avoid it https://elastalert2.readthedocs.io/en/latest/recipes/faq.html?highlight=days#how-can-i-prevent-duplicate-alerts
but maybe there is some way to filter with a query but I couldn't find it.
These are my rules:

rules:
  cpu_usage_up_period_60_output: |-
    ---
    name: CLUSTER CPU USAGE UP PERIOD_60
    type: metric_aggregation
    index: metricbeat-*
    metric_agg_key: system.cpu.total.norm.pct
    metric_agg_type: avg
    query_key: bucketID
    min_doc_count: 1
    doc_type: _doc
    buffer_time:
      hours: 1
    bucket_interval:
      minutes: 60
    sync_bucket_interval: true
    max_threshold: 0.8
    filter:
    - term:
        metricset.name: cpu
    alert:
    - "slack"
    - "post"
    slack:
    slack_webhook_url: "https://hooks.slack.com/services/xxxxxxxxxxxxxxxxxxxxxxxxxx"
    post:
    http_post_url: "http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    http_post_static_payload:
      infra_component: "kubernetes"
      operation: "up"
    http_post_payload:
      bucket_id: bucketID
    realert:
      minutes: 1
  cpu_usage_down_period_60_output: |-
    ---
    name: CLUSTER CPU USAGE DOWN PERIOD_60
    type: metric_aggregation
    index: metricbeat-*
    metric_agg_key: system.cpu.total.norm.pct
    metric_agg_type: avg
    query_key: bucketID
    min_doc_count: 1
    doc_type: _doc
    buffer_time:
      hours: 1
    bucket_interval:
      minutes: 60
    sync_bucket_interval: true
    min_threshold: 0.4
    filter:
    - term:
        metricset.name: cpu
    alert:
    - "slack"
    - "post"
    slack:
    slack_webhook_url: "https://hooks.slack.com/services/XxXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    post:
    http_post_url: "http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    http_post_static_payload:
      infra_component: "kubernetes"
      operation: "down"
    http_post_payload:
      bucket_id: bucketID
    realert:
      days: 1

Regards!

[jertel]

Is the Kubernetes Cluster Autoscaler not an option for your deployment? It handles cluster level scaling for you, based on a number of inputs, including CPU usage. The major cloud providers include this feature.

Assuming you have a reason to not use the autoscaler, one option with ElastAlert 2 is to have the rule disable itself once it triggers, and then when the 80% rule triggers, have it re-enable the 20% rule. To make a rule disable itself, and/or enable another rule, would require creating an enhancement that performs this work. It would flip the rule's is_enabled key value to true or false as needed.

Drawbacks of this approach is that you will struggle to keep the scale state synchronized with the rules. Ex: Someone manually scales up the cluster, but ElastAlert 2 is unaware and thinks it's still all the way scaled down. To account for this you would need to go the extra step of having the enhancement query the kubernetes cluster API to learn about the current scale state.

I recommend that you try to find a way to use the Cluster Autoscaler.

[masterplanx]

Hi @jertel! yeah, we use autoscaler, but we have an API that we need to send information about the limit per user (user in the platform) so we change the max nodes (in DO), etc, so is more complex.
So we use http_post for that and is working well, just we didn't find the proper way maybe to set a filter to avoid the min_threshold all the time. (for example, we know that the minimum is 3 nodes), is that more clear? What do you think?

About the option we can try I think.
Regards!!