Parsing IIS logs (W3C log file format)

[yavulan]

Given log file content is:

#Software: Microsoft Internet Information Services
#Version: 1.0
#Date: 2023-02-23 00:00:01
#Fields: date time
2023-02-23 00:00:02
2023-02-23 00:00:03
#Software: Microsoft Internet Information Services
#Version: 1.0
#Date: 2023-02-24 00:00:01
#Fields: date cs-method
2023-02-24 GET
2023-02-24 POST

Do you see an approach to dynamically detect structure and transform it into messages like this:

{"date": "2023-02-23", "time": "00:00:02"}
{"date": "2023-02-23", "time": "00:00:03"}
{"date": "2023-02-24", "cs-method": "GET"}
{"date": "2023-02-24", "cs-method": "POST"}

Thank you in advance!

[jszwedko]

Hi @yavulan ,

It won't be straight-forward, but it seems like this should be possible through configuration of the multiline options on the file source to group each entry together and then parsing it. It seems like:

#Software: Microsoft Internet Information Services
#Version: 1.0
#Date: 2023-02-23 00:00:01
#Fields: date time
2023-02-23 00:00:02
2023-02-23 00:00:03

Would be one entry. You could then parse that that emit multiple events from a remap program following https://vector.dev/highlights/2021-07-16-remap-multiple/. If VRL proves insufficient for parsing, you could also use a lua transform.

If it is possible to change the output format to something more structured, though, I'd recommend doing that since parsing that sort of data is not going to be straight-forward, as you can see.

[yavulan]

Thank you for answering so fast!