Support derivative aggregations on alerting for incremental data

[archfz]

Given metricbeat and it's network metricset we receive the total number of bytes in and out for each document. There is not delta provided. Unfortunately metricbeat is not going to implement non incremental value for this metricset: elastic/beats#2783

Now from what I have read the derivative aggregation would be required to be able to set a rule that says alert if this many bytes per second we have in the last timeperiod. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-derivative-aggregation.html#search-aggregations-pipeline-derivative-aggregation

See example document system.network.in.bytes and system.network.out.bytes. These fields are only increasing from doc to doc, they don't represent deltas, they represent totals. Now from the possible rules for elastalert2 as I see none of them fit to analyse for this type of data.

{
  "_index": "metricbeat-7.11.2-2021.05.06-000006",
  "_type": "_doc",
  "_id": "N8lhS3kBTvEe6jk05x8z",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-05-08T09:49:36.968Z",
    "system": {
      "network": {
        "name": "eth2",
        "in": {
          "bytes": 799223433,
          "packets": 618846,
          "errors": 0,
          "dropped": 0
        },
        "out": {
          "dropped": 0,
          "packets": 618255,
          "bytes": 65183723,
          "errors": 0
        }
      }
    },
    "ecs": {
      "version": "1.7.0"
    },
    "host": {
      "ip": [
        "172.18.0.2",
        "172.19.0.3",
        "172.20.0.10"
      ],
      "mac": [
        "02:42:ac:12:00:02",
        "02:42:ac:13:00:03",
        "02:42:ac:14:00:0a"
      ],
      "hostname": "10ea8b886435",
      "architecture": "x86_64",
      "os": {
        "version": "10 (buster)",
        "family": "debian",
        "name": "Debian GNU/Linux",
        "kernel": "4.15.0-140-generic",
        "codename": "buster",
        "platform": "debian"
      },
      "id": "888b2975747fbae801ada88be3481c09",
      "containerized": true,
      "name": "master-proxy"
    },
    "agent": {
      "ephemeral_id": "4bdc750f-16df-4714-b4b9-122f31326573",
      "id": "878038ed-5646-4915-9447-540c8a8529e6",
      "name": "master-proxy",
      "type": "metricbeat",
      "version": "7.11.2",
      "hostname": "10ea8b886435"
    },
    "event": {
      "duration": 793471,
      "dataset": "system.network",
      "module": "system"
    },
    "metricset": {
      "name": "network",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-05-08T09:49:36.968Z"
    ]
  },
  "highlight": {
    "metricset.name": [
      "@kibana-highlighted-field@network@/kibana-highlighted-field@"
    ],
    "system.network.name": [
      "@kibana-highlighted-field@eth2@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1620467376968
  ]
}

[jertel]

I don't think I've attempted to do this in any of my rules so I can't offer much help on the ElastAlert side. But if you're a developer you could look into writing a plugin for your beats, logstash, or ingest pipeline that would always retain the previous byte count and calculate the bps value for a new data field. It's probably not the answer you were hoping for since it is more involved, but it should work.

[jertel]

Converting to discussion due to lack of follow-up by OP, and no apparent interest from the remaining ElastAlert 2 community.