understand rule testing (flatline)

[mansior]

Hello,

I would like to be alerted as soon as an elastic document does not occur for 30 minutes. This should be possible with type: flatline

My current rule is:

name: test_es_index---no_successfully_within_30min
type: flatline
index: test_es_index
timestamp_field: timestamp
use_count_query: true
doc_type: _doc
threshold: 1
timeframe:
  minutes: 30
filter:
- query:
    query_string:
      query: "msg:(+'Successfully sent result to Destination' +'Status-Code' +'200')"

looks really simple, therefor let us test it:

$ elastalert-test-rule test_es_index-no_successfully_within_30min.yaml
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
Got 164 hits from the last 0 day

Available terms in first hit:
	timestamp
	msg
	type

INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:15 UTC to 2021-11-04 14:16 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:16 UTC to 2021-11-04 14:17 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:17 UTC to 2021-11-04 14:18 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:18 UTC to 2021-11-04 14:19 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:19 UTC to 2021-11-04 14:20 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:20 UTC to 2021-11-04 14:21 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:21 UTC to 2021-11-04 14:22 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:22 UTC to 2021-11-04 14:23 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:23 UTC to 2021-11-04 14:24 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:24 UTC to 2021-11-04 14:25 UTC: 1 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:25 UTC to 2021-11-04 14:26 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:26 UTC to 2021-11-04 14:27 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:27 UTC to 2021-11-04 14:28 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:28 UTC to 2021-11-04 14:29 UTC: 2 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:29 UTC to 2021-11-04 14:30 UTC: 3 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:30 UTC to 2021-11-04 14:31 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:31 UTC to 2021-11-04 14:32 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:32 UTC to 2021-11-04 14:33 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:33 UTC to 2021-11-04 14:34 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:34 UTC to 2021-11-04 14:35 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:35 UTC to 2021-11-04 14:36 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:36 UTC to 2021-11-04 14:37 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:37 UTC to 2021-11-04 14:38 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:38 UTC to 2021-11-04 14:39 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:39 UTC to 2021-11-04 14:40 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:40 UTC to 2021-11-04 14:41 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:41 UTC to 2021-11-04 14:42 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:42 UTC to 2021-11-04 14:43 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:43 UTC to 2021-11-04 14:44 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:44 UTC to 2021-11-04 14:45 UTC: 0 hits
INFO:elastalert:Queried rule test_es_index---no_successfully_within_30min from 2021-11-04 14:45 UTC to 2021-11-04 14:45 UTC: 1 hits

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'test_es_index---no_successfully_within_30min', 'endtime': datetime.datetime(2021, 11, 4, 14, 45, 25, 834477, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 11, 4, 14, 15, 7, 834477, tzinfo=tzutc()), 'matches': 0, 'hits': 7, '@timestamp': datetime.datetime(2021, 11, 4, 14, 45, 28, 730829, tzinfo=tzutc()), 'time_taken': 2.0935797691345215}

Got 164 hits from the last 0 day => count is correct, the filter seems to work.

But why is:

• the total timeframe (or range) 30 minutes instead of 1 day (2021-11-04 14:15 UTC to 2021-11-04 14:45 UTC)
• a single timeframe 1 minute instead of 30 minutes?

Am I fundamentally misunderstanding the whole thing?

Thanks

[jertel]

It's not clear to me what the problem or question is here. The rule must run for the entire timeframe before it can alert on a flatline. Meanwhile, after each minute the rule is executed, which is why you see the 1 minute log lines, yet no alert. Once it hits the full 30 minutes you then see the final log output.

[mansior]

Thank you, I understand now. elastalert-test-rule test the whole configuration, including the parameters from config.yaml