-
|
My use-case is to create metrics from logs and aggregate them. Vector's log_to_metric + aggregate transforms doesn't seem to use event's timestamp. Config: [log_schema]
timestamp_key = "@timestamp"
[sources.file_input]
type = "file" # required
include = ["/logs/*.json"] # required
read_from = "beginning"
[transforms.parse_log_json]
type = "remap"
inputs = ["file_input"]
source = '''
. = parse_json!(string!(.message))
'''
#####################
[transforms.log_to_met]
type = "log_to_metric" # required
inputs = [ "parse_log_json" ] # required
[[transforms.log_to_met.metrics]]
type = "counter"
field = "statuscode"
name = "response_total"
[transforms.log_to_met.metrics.tags]
statuscode = "{{statuscode}}"
[transforms.msg_agg]
type = "aggregate"
inputs = [ "log_to_met" ]
interval_ms = 5_000
[sinks.console]
type = "console"
inputs = [ "msg_agg" ]
target = "stdout"
[sinks.console.encoding]
codec = "json"Input file: Output: Expected output: |
Beta Was this translation helpful? Give feedback.
Answered by
tobz
Aug 16, 2022
Replies: 1 comment 1 reply
-
|
Chatting with @fpytloun in Discord, we talked through the intended/expected behavior, which is actually windowing the metrics by their own timestamp, not the real clock time as Vector processes the event. Thus, the goal would be to aggregate each metric individually with a 5 second window, based on the original timestamp... and thus, with the example input, having timestamps more than 5 seconds apart, the expectation effectively has no reduction/aggregation. With the intended behavior explained here, the answer is that At present, the best approach would likely be to use the |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thank you for explaining this. I'll try to use lua for my use-case. Also created follow-up feature request: #14015 |
Beta Was this translation helpful? Give feedback.
Answer selected by
jszwedko
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Chatting with @fpytloun in Discord, we talked through the intended/expected behavior, which is actually windowing the metrics by their own timestamp, not the real clock time as Vector processes the event. Thus, the goal would be to aggregate each metric individually with a 5 second window, based on the original timestamp... and thus, with the example input, having timestamps more than 5 seconds apart, the expectation effectively has no reduction/aggregation.
With the intended behavior explained here, the answer is that
aggregatetransform operates on wall clock time, and so it has no concept of being able to window incoming data in this way. Similarly, thereducetransform gets a little c…