Assistance with Decoding Multi-Line Logs Using Vector

[Giampearo]

Hi everyone,

1. I've been working on decoding structured logs and encountered difficulties due to their multi-line format. Below is an example of the log format I'm dealing with:

{
    "parameters": "ERR (!!): Src:GOTREQ-CreateCharge Msg:{\"header\":{\"timestamp\":\"2025-01-24T09:16:11Z\"},\"body\":{\"orderid\":\"1234567890\",\"customername\":\"JOHN DOE\",\"orderstatus\":\"2\",\"orderstatusdesc\":\"Pending Inventory\",\"email\":\"example@email.com\",\"callbackinforar\":\"0\",\"callbackinforoms\":\"1\",\"faid\":\"9876543210\",\"baid\":\"123456789\",\"dealercode\":\"A1234.56789\",\"submissiondate\":\"20250124171404\",\"paymentItems\":[{\"serviceid\":\"60123456789\",\"price\":\"0.0\",\"articleid\":\"100016933001\",\"uomid\":\"2AD\",\"discountid\":\"\",\"discountamount\":\"\",\"prepostind\":\"POSTPAID\",\"accountcatdesc\":\"C\",\"marketcodedesc\":\"H\",\"itemno\":\"3491715875\",\"itemtaxgroupid\":\"0\",\"taxamount\":\"0\",\"taxpercentage\":\"0\",\"billinginvoiceno\":\"\",\"custSegment\":null},{\"serviceid\":\"60123456789\",\"price\":\"0.0\",\"articleid\":\"80000826\",\"uomid\":\"EA\",\"discountid\":\"\",\"discountamount\":\"\",\"prepostind\":\"POSTPAID\",\"accountcatdesc\":\"C\",\"marketcodedesc\":\"H\",\"itemno\":\"3491715874\",\"itemtaxgroupid\":\"0\",\"taxamount\":\"0\",\"taxpercentage\":\"0\",\"billinginvoiceno\":\"\",\"custSegment\":null}],\"misc01\":null,\"misc02\":null,\"misc03\":null,\"misc04\":null,\"misc05\":null}} Ref:"
}

2. What I Have Tried So Far Using the VRL Playground: I implemented the following Vector configuration for parsing:

parsed = parse_regex!(.parameters, r'^(?P<severity>\w+ \(\!\!\)):\sSrc:(?P<source>[\w-]+)\sMsg:(?P<json_payload>\{.*\})\sRef:$', true)

.severity = get(parsed, ["severity"]) ?? ""
.source = get(parsed, ["source"]) ?? ""
.parameters_json = get(parsed, ["json_payload"]) ?? ""

if exists(.parameters_json) {
    .structured_data = parse_json!(.parameters_json)  # Removed `?? {}` since parse_json!() never fails
    . = merge!(., .structured_data)
}

del(.parameters_json)
del(.message)
del(.structured_data)

3. Expected Output: I expect the parsed output to be structured as follows:

{ "severity": "ERR (!!)", "source": "GOTREQ-CreateCharge", "header": { "timestamp": "2025-01-24T09:16:11Z" }, "body": { "orderid": "1234567890", "customername": "JOHN DOE", "orderstatus": "2", "orderstatusdesc": "Pending Inventory", "email": "example@email.com", "callbackinforar": "0", "callbackinforoms": "1", "faid": "9876543210", "baid": "123456789", "dealercode": "A1234.56789", "submissiondate": "20250124171404", "paymentItems": [ { "serviceid": "60123456789", "price": "0.0", "articleid": "100016933001", "uomid": "2AD", "discountid": "", "discountamount": "", "prepostind": "POSTPAID", "accountcatdesc": "C", "marketcodedesc": "H", "itemno": "3491715875", "itemtaxgroupid": "0", "taxamount": "0", "taxpercentage": "0", "billinginvoiceno": "", "custSegment": null }, { "serviceid": "60123456789", "price": "0.0", "articleid": "80000826", "uomid": "EA", "discountid": "", "discountamount": "", "prepostind": "POSTPAID", "accountcatdesc": "C", "marketcodedesc": "H", "itemno": "3491715874", "itemtaxgroupid": "0", "taxamount": "0", "taxpercentage": "0", "billinginvoiceno": "", "custSegment": null } ] } }

4. Issue Encountered: The configuration works in the VRL Playground but fails when applied in Vector.toml for actual log processing.
   The actual logs contain multiple entries, making it hard to extract and structure the data properly.
   Below is an example of how the logs appear in the system:

[ { "_index": "createcharge_log_info-2025.01.24", "_type": "createcharge_log_info_type", "_id": "createcharge_log_info-2025.01.24+2+1234567", "_score": 2.0, "_source": { "environment": "Production", "messagesource": "CreateCharge", "logdatetime": "2025-01-24T09:16:11.9655377Z", "message": "Log Message", "parameters": "ERR (!!): Src:GOTREQ-CreateCharge Msg:{...} Ref:" } }, { "_index": "createcharge_log_info-2025.01.24", "_type": "createcharge_log_info_type", "_id": "createcharge_log_info-2025.01.24+0+9876543", "_score": 2.0, "_source": { "environment": "Production", "messagesource": "CreateCharge", "logdatetime": "2025-01-24T09:21:20.379127Z", "message": "Log Message", "parameters": "ERR (!!): Src:ProcessingService>>ProcessHandler Msg:Start proceed create charge handle for order: 9876543210 Ref:" } } ]

5. What I Need Help With Handling Multi-Line Logs:

a. How can I modify my Vector.toml config to correctly process multi-line logs?
The logs appear as nested JSON inside the parameters field, making it tricky to extract and structure properly.
Efficient Parsing for Multiple Entries:

b. How can I ensure that each log entry is processed individually without losing its structure?
Best Approach for Structured Fields:

c. Are there any alternative methods or regex optimizations that could improve field extraction in Vector?
Any help or insights would be greatly appreciated! Thanks in advance.

[pront]

Hi @Giampearo, what's your source? How do you decode the bytes as they come in to Vector? It seems like you are dealing with a valid JSON array which should be decoded first and then you can call parse_json on inner strings.

[Giampearo]

The logs I'm working with are extracted from a Kibana server, stored on a server, and then processed using Vector to flatten them before forwarding them to Loki.

The issue is that the log format is inconsistent:

Sometimes it's plain text + nested JSON.
Other times, it’s just nested JSON.
Occasionally, it's pure plain text, but in different formats, like

Msg: XYZ rr: xyz

Additionally, the logs can have deeply nested structures, e.g., [[{}]]—a nested array containing JSON objects and raw text. Can Vector decode and normalize these logs first before processing them? How would VRL handle cases where we need to extract structured JSON while also parsing mixed plain text?

[pront]

I see, the source can decode them as bytes and then Remap can be used to process them with multiple attempts. I am not sure how complicated the VRL source will be, but it is possible to output multiple events.