limit_execution: delayed notifications

[mkondratev]

I have the following rules with different limit_execution times.

name: MonFriWorkTime
type: flatline
limit_execution: "* 6-19 * * 1-5"
threshold: 1
timeframe:
  minutes: 10
  
name: MonFriNonWorkTime
type: flatline
limit_execution: "* 0-5,20-23 * * 1-5"
threshold: 1
timeframe:
  minutes: 30

name: SatSun
type: flatline
limit_execution: "* * * * 6,0"
threshold: 1
timeframe:
  minutes: 30

I expect to receive alerts within the time interval specified in limit_execution about events that occurred during limit_execution. In reality, the rule triggers outside of limit_execution and sends an alert within limit_execution, meaning after a 6-hour delay.

{
  "_index": "elastalert",
  "_type": "_doc",
  "_id": "OA-FoJcBWBDl74cr2ge3",
  "_score": 0.0,
  "_source": {
    "match_body": {
      "@timestamp": "2025-06-23T23:57:23.663253Z",
      "key": "all",
      "count": 0,
      "threshold": 1,
      "num_hits": 330,
      "num_matches": 2
    },
    "rule_name": "MonFriWorkTime",
    "alert_info": {
      "type": "alertmanager"
    },
    "alert_sent": true,
    "alert_time": "2025-06-24T06:00:16.237305Z",
    "match_time": "2025-06-23T23:57:23.663253Z",
    "@timestamp": "2025-06-24T06:00:16.309467Z"
  }
}

This results in delayed notifications. How can I fix this?

[jertel]

limit_execution pauses a rule from running outside of the configured execution time. That's it.

Therefore, when the rule is allowed to run again, it will continue where it left off. The purpose of this is to stop people from getting alerts outside of business hours, not to completely ignore triggerable events that occurred outside of the execution window. It does not trigger outside of the execution window, but yes, once it runs again it indeed can still find those matching events that had previously occurred.

To completely ignore events that occurred outside of the execution window you might be able to specific that in the filter. But I haven't tried it and you'd have to read up on that range filter syntax.

[mkondratev]

So, I need to implement limit_execution via Query DSL or add filtering in enhancements to raise DropMatchException() for events outside the time window.

[jertel]

limit_execution is already implemented and works as I mentioned above. It limits when the rule is executed.

The query filter would be used filter out undesired events, by time (assuming it's possible -- again, haven't tried it).

Your enhancement idea seems like an alternative approach that could work as well to perform the filtering.

[nsano-rururu]

https://github.com/0xSeb/elastalert_hour_range/blob/master/elastalert_modules/hour_range_enhancement.py

[mkondratev]

pauses a rule from running outside of the configured execution time.

Actually, that's not quite accurate. For limit_execution: "* 6-19 * * 1-5" and "match_time": "2025-06-23T23:57:23.663253Z", the rule runs outside the specified window since the cron allows execution every minute between 6 and 19 UTC, but the match_time is 23:57:23 UTC (due to the "Z" timezone marker).

elastalert@elastalert2-7c95bc5cb7-f2f8d:~$ date
Wed Jun 25 07:29:58 UTC 2025

elastalert@elastalert2-7c95bc5cb7-f2f8d:~$ elastalert-test-rule --config=config.yaml rules/MonFriWorkTime.yaml --start="2025-06-23 23:40" --end="2025-06-24 00:20"
Got 7 hits from the last 0 day
INFO:elastalert:Queried rule MonFriWorkTime from 2025-06-23 23:40 UTC to 2025-06-23 23:50 UTC: 3 / 3 hits
INFO:elastalert:Queried rule MonFriWorkTime from 2025-06-23 23:50 UTC to 2025-06-24 00:00 UTC: 0 / 0 hits
INFO:elastalert:Queried rule MonFriWorkTime from 2025-06-24 00:00 UTC to 2025-06-24 00:10 UTC: 2 / 2 hits
INFO:elastalert:Queried rule MonFriWorkTime from 2025-06-24 00:10 UTC to 2025-06-24 00:20 UTC: 2 / 2 hits
INFO:elastalert:Alert for MonFriWorkTime at 2025-06-24T00:00:00Z:

[mkondratev]

If limit_execution restricts the rule's execution time to the specified schedule, why does the metadata index contain a record of an event detected outside this schedule?

"alert_time": "2025-06-24T06:00:16.237305Z",
> "match_time": "2025-06-23T23:57:23.663253Z",
"@timestamp": "2025-06-24T06:00:16.309467Z"

The match_time is not between 2025-06-23T06:00:00.000000Z and 2025-06-23T19:59:59.999999Z.

[jertel]

From elastalert.py::1472:

        match_time = lookup_es_key(match, rule['timestamp_field'])
        if match_time is not None:
            body['match_time'] = match_time

match_time is a copy of the triggering event's @timestamp field which can be any time. It is not the time when ElastAlert2 processed/ran a rule.

[mkondratev]

What is then recorded in the metadata index, if not the actual event?

[jertel]

I'm not sure what you're asking. You already pasted a copy of what's recorded in that elastalert metadata index at the start of this discussion. Are you asking generally what that data represents? If so, it's the name of the rule that triggered an alert, the info about the match itself, the info about the alert that was sent, when the alert was sent, when the metadata record was added, etc.

[mkondratev]

I'm trying to figure out what triggered the alert.

elastalert - is a log of information about every alert triggered and contains.

An alert is sent if the threshold is less than 1. However, the threshold is greater than 1 between 06:00 and 19:59.

match_body: This is the contents of the match dictionary that is used to create the alert. The subfields may include a number of things containing information about the alert.

    "match_body": {
      "@timestamp": "2025-06-23T23:57:23.663253Z",
      "key": "all",
      "count": 0, < count of search results?
      "threshold": 1,
      "num_hits": 330,
      "num_matches": 2 < matches of threshold < 1?
    }

[mkondratev]

Enhancements:

import datetime

from croniter import croniter
from elastalert.enhancements import BaseEnhancement, DropMatchException


class LimitExecutionEnhancement(BaseEnhancement):
    def process(self, match):
        limit = self.rule.get('limit_execution')
        if not limit:
            return

        ts = match.get('@timestamp')
        if not ts:
            return

        event_time = datetime.datetime.fromisoformat(ts.replace("Z", "+00:00"))

        if not croniter.match(limit, event_time):
            raise DropMatchException()