Using parse_json with unicode in message

[fpytloun]

Is there a way how to handle unicode when parsing JSON with parse_json function like this:

msg_parsed, err = parse_json(string!(.message))

If JSON contains unicode in field value, eg.

\"req_body\":\"��\\u0000\\u0005\"

parsing will fail with:

function call error for \"parse_json\" at (20:49): unable to parse json: invalid unicode code point at line 1 column 8587

Message is read from kafka.
Any idea how to properly escape or url-encode message before parsing or some other way?

[fpytloun]

This seems to be the solution 🤔

      msg_parsed, err = parse_json(replace(.message, r'([^\x00-\x7F])', "\\\\$$1") ?? .message)

[jszwedko]

Hey @fpytloun ! Can you provide the raw message as a file? That would help testing this out.

[pablosichert]

Heyo @fpytloun, I tried poking at this locally with the following example in vector vrl:

$ parse_json!("{ \"foo\": \"😃\\u2665\" }")
{ "foo": "😃♥" }

It looks like using unicode characters itself is not the problem, rather than that your string contains malformed/invalid unicode?

[fpytloun]

I've hit the same issue when using HTTP source and ingesting logs from Fluentbit using http output. Fluentbit will get 400 Bad request and it's not retryable:

`{"code":400,"message":"Failed decoding body: ParsingError(Error parsing JSON: Error(\"invalid unicode code point\", line: 1, column: 368341))"}`

Unfortunately I don't have reproducer so far but it's clear that there must be something wrong with way how Vector decodes JSON with unicode garbage (no matter if it's from Kafka or any other source).

Config is very simple:

    [sources.in_fluent]
      type = "http"
      address = "0.0.0.0:24280"
      encoding = "json"

      tls.enabled = true
      tls.verify_certificate = true
      tls.ca_file = "/secrets/identity/client_ca_with_fluent.crt"
      tls.crt_file = "/secrets/identity/server.crt"
      tls.key_file = "/secrets/identity/server.key"

[fpytloun]

Can be workarounded on Fluentbit side by escaping unicode characters with Lua:

function deep_replace(obj, source, target)
   if obj then
      if type(obj) == "table" then
         local result = {}
         for k,v in pairs(obj) do
            result[k] = deep_replace(v, source, target)
         end
         return result
      elseif type(obj) == "string" then
         return obj:gsub(source, target)
      else
         return obj
      end
   end

   return obj
end

function escape_non_ascii(tag, timestamp, record)
   local new_record = deep_replace(record, "([\128-\255])", "\\\\%1")
   return 1, timestamp, new_record
end

and config:

    [FILTER]
        Name    lua
        Match   *
        Script  /fluent-bit/scripts/filters.lua
        Call    escape_non_ascii