multiline support in vrl

[LinTechSo]

Hi all.
Can we specify a multiline parser in the transform for multi line logs ?
something like https://docs.fluentd.org/parser/multiline
Thanks.

[Sjoertjuh]

You should be able to parse multiline logs by using the multiline option in the source, and the multiline regex flag. Here is an example config (in yaml) on how it can work:

sources:
  test_source:
    type: file
    include:
      - /var/log/test.log
    multiline:
      start_pattern: '^\[\d+-\d+-\d+ \d+:\d+:\d+'
      mode: halt_before
      condition_pattern: '^\[\d+-\d+-\d+ \d+:\d+:\d+'
      timeout_ms: 1000

transforms:
  test_transform:
    type: remap
    inputs:
      - test_source
    source: | #                      \/ This flag makes regex multiline.
      . |= parse_regex!(.message, r'(?m)^\[(?P<timestamp>\d+-\d+-\d+ \d+:\d+:\d+)] This is a (?P<type>\w+) (?P<severity>\w+)\n(?P<message>(.|\n)*)')

tests:
  -
    name: multiline test
    inputs:
      -
        insert_at: test_transform
        value: "[2023-12-31 12:00:00] This is a multiline error\nIt has newlines without spaces...\n  and with spaces"
    outputs:
      -
        extract_from: test_transform
        conditions:
          -
            type: vrl
            source: |
              assert_eq!(.message, "It has newlines without spaces...\n  and with spaces")

              assert_eq!(.severity, "error")

              assert_eq!(.timestamp, "2023-12-31 12:00:00")

              assert_eq!(.type, "multiline")

Check the docs for more info on the multiline file source option.
There are also some common examples at the bottom of the page