How can I aggregate logs based on fields

[nicolasmf]

Hello,
I'm trying to write a configuration to aggregate logs received by UDP by devname, srcip, dstip and dstport fields.

The logs I'm working on are Fortinet FW logs.

Example log :
Feb 1 13:12:34 <Redacted_IP> <180>date=2023-01-23 time=13:57:22 devname="SOME_DEVICE_NAME" devid="..." logid="..." type="..." subtype="..." level="..." vd="..." eventtime=... tz="..." srcip=1.2.3.4 srcport=1234 srcintf="..." srcintfrole="..." dstip=1.2.3.4 dstport=1234 dstintf="..." dstintfrole="..." sessionid=... proto=... action="..." policyid=... policytype="..." poluuid="..." service="..." dstcountry="..." srccountry="..." trandisp="..." duration=... sentbyte=... rcvdbyte=... sentpkt=... rcvdpkt=... appcat="..." sentdelta=... rcvddelta=...

If, in a 10 seconds interval, 8 logs have the same devname, srcip, dstip and dstport, this should be outputed :

Feb 1 13:12:34 <Redacted_IP> <180>date=2023-01-23 time=13:57:22 devname="SOME_DEVICE_NAME" devid="..." logid="..." type="..." subtype="..." level="..." vd="..." eventtime=... tz="..." srcip=1.2.3.4 srcport=1234 srcintf="..." srcintfrole="..." dstip=1.2.3.4 dstport=1234 dstintf="..." dstintfrole="..." sessionid=... proto=... action="..." policyid=... policytype="..." poluuid="..." service="..." dstcountry="..." srccountry="..." trandisp="..." duration=... sentbyte=... rcvdbyte=... sentpkt=... rcvdpkt=... appcat="..." sentdelta=... rcvddelta=... log_count=8

How can I do that ? I tried using the reduce transformation, but the problem is that when I receive 3 same logs and 1 unique log, it outputs only one log, based on the first 3.

Thank you,

[StephenWakely]

Dedupe almost does what you want. The only thing is it won't append a count to the output log message.

Short of falling back on Lua there isn't a way to do exactly what you want. Would you be able to add a feature request (https://github.com/vectordotdev/vector/issues) detailing what you need and if there is sufficient demand for it we can take a look.

[nicolasmf]

Hey Stephen,

The issue I have with dedupe is that I can't flush the cache. I'd like to be able to flush the cache (every 10 seconds in my case).

I'll open a feature request, thanks !