Parse Grok after Multiline event capture truncates the multiline event.

[atibdialpad]

I have a Python Stacktrace that I am capturing from the log file using the 'file ' source.
The trace looks like:

2021-12-28 07:21:30,793 ERROR
Traceback (most recent call last):
  File "/usr/local/rackman/lib/python2.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/rackman/lib/python2.7/site-packages/flask/app.py", line 1926, in dispatch_request
    self.raise_routing_exception(req)                                                                                                                      
  File "/usr/local/rackman/lib/python2.7/site-packages/flask/app.py", line 1908, in raise_routing_exception
    raise request.routing_exception
NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

The multi-line config looks like this. Basically says "to group all lines that don't start with a TIMESTAMP with the last line that did in a single event"

multiline.start_pattern = '.*'
multiline.condition_pattern = '^(?P<year>[0-9]{4})-(?P<month>[0-9]{2})-(?P<day>[0-9]{2})'
multiline.mode = "halt_before"

Using this, I am able to capture the event successfully, the event looks like this :

{"file":"/Users/atib/src/VECTOR/test/rackman.log","host":"mdatib.local","message":"2021-12-28 07:21:30,793 ERROR\nTraceback (most recent call last):\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1950, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1926, in dispatch_request\n    self.raise_routing_exception(req)\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1908, in raise_routing_exception\n    raise request.routing_exception\nNotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","service":{"name":"rackman"},"source_type":"file","timestamp":"2021-12-29T04:07:36.814126Z"}

However when I pass it through another GROK layer (to extract to loglevel, date, exception type, etc..) I loose the date after the newline.

pattern = s'%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{GREEDYDATA:message}'
match, err = parse_grok(.message, pattern)

post which my event looks like

{"file":"/Users/atib/src/VECTOR/test/rackman.log","host":"mdatib.local","loglevel":"ERROR","message":"Traceback (most recent call last):","service":{"name":"rackman"},"source_type":"file","timestamp":"2021-12-28 07:21:30,793"}

As you can see, the message filed is reduced to only the first line of the multi-line event. I guess this is something to do with escaping \n in the msg field but since the msg is generated internally in the pipeline, Vector should handle this ??

@jszwedko

[atibdialpad]

could be avoided if we could treat message as a raw string. Either a config to ingest log lines as Raw string or a remap transform to convert a string to raw string.

[jszwedko]

Hi @atibdialpad !

It looks like you can use regex classes to enable multi-line matching behavior:

$ . = {"file":"/Users/atib/src/VECTOR/test/rackman.log","host":"mdatib.local","message":"2021-12-28 07:21:30,793 ERROR\nTraceback (most recent call last):\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1950, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1926, in dispatch_request\n    self.raise_routing_exception(req)\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1908, in raise_routing_exception\n    raise request.routing_exception\nNotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","service":{"name":"rackman"},"source_type":"file","timestamp":"2021-12-29T04:07:36.814126Z"}
{ "file": "/Users/atib/src/VECTOR/test/rackman.log", "host": "mdatib.local", "message": "2021-12-28 07:21:30,793 ERROR\nTraceback (most recent call last):\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1950, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1926, in dispatch_request\n    self.raise_routing_exception(req)\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1908, in raise_routing_exception\n    raise request.routing_exception\nNotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "service": { "name": "rackman" }, "source_type": "file", "timestamp": "2021-12-29T04:07:36.814126Z" }

$ pattern = s'(?m)%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{GREEDYDATA:message}'

"(?m)%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{GREEDYDATA:message}"

$ match, err = parse_grok(.message, pattern)

{ "loglevel": "ERROR", "message": "Traceback (most recent call last):\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1950, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1926, in dispatch_request\n    self.raise_routing_exception(req)\n  File \"/usr/local/rackman/lib/python2.7/site-packages/flask/app.py\", line 1908, in raise_routing_exception\n    raise request.routing_exception\nNotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "timestamp": "2021-12-28 07:21:30,793" }

It looks like GREEDYDATA is just translated to .* which won't match newlines by default.

[hoangphuocbk]

Hi @jszwedko,

I encountered the same problem but I used parse_regex function with VRL. How can I specify (?m) in the multiline case?

[jszwedko]

Hi @hoangphuocbk !

You can pass it into the pattern like: parse_regex(.message, r'(?m)some pattern').

[hoangphuocbk]

I already tried this pattern, but it's not working.
Here are my sample data:

12/05/2022 11:08:31 INFO  http-nio-8812-exec-9: commitCurrentSessions(): Commited transaction!
12/05/2022 11:08:31 INFO  http-nio-8812-exec-9: Session is null or closed
12/05/2022 11:08:31 INFO  http-nio-8812-exec-9: RESPONSE FROM API
  <field id="0" value="0500"/>
  <field id="3" value="PT033"/>
12/05/2022 11:08:31 INFO  http-nio-8812-exec-9: Check GetFee

I tried to parse these data with

sources:
  sample_input:
    type: file
    include:
      - /etc/vector/test.log
    multiline:
      start_pattern: '^\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}'
      condition_pattern: '^\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}'
      mode: 'halt_before'
      timeout_ms: 1000

transforms:
  sample_transforms:
    type: remap
    inputs:
      - sample_input
    source: >-

      . |= parse_regex!(.message, r'(?m)^(?P<timestamp>\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2})\s+(?P<severity>\w+)\s+(?P<message>.*)$')

      # Coerce parsed fields

      .timestamp = parse_timestamp(.timestamp, "%d/%m/%Y %H:%M:%S") ?? now()

However, I only get "http-nio-8812-exec-9: RESPONSE FROM API" as message in the multiline example. May I miss somethin?

[jszwedko]

Hi @hoangphuocbk !

By default, . won't match newlines. You can supply the extra (?s) modifier to do this. So, something like:

      . |= parse_regex!(.message, r'(?sm)^(?P<timestamp>\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2})\s+(?P<severity>\w+)\s+(?P<message>.*)$')

You can see a description of all of the modifiers in https://docs.rs/regex/latest/regex/#grouping-and-flags