Chunked GELF: Support for Non-Chunked Messages and Mixed Compression Methods

[arch-xtof]

Hello, I'm excited about Vector's upcoming support for chunked GELF messages! This feature will simplify our log pipeline quite a lot. I know this has not been released yet, but I already have a couple of questions regarding how Vector will handle different types of GELF messages.

In my environment, we have multiple implementations for sending GELF messages, and logs vary quite a bit. I’d like to know the following:

1. Will I still be able to process non-chunked messages when using framing.method = "chunked_gelf"?
2. Will this support chunks with mixed compression methods (gzip, zlib, or uncompressed)?

I know the implementation is similar to the go-gelf specification (which supports these), but since I don't understand Rust I can't confirm this from the source code.

Thanks in advance for your help!

[jszwedko]

Awesome, glad to hear you will find this feature useful @arch-xtof ! Credit to @jorgehermo9 for implementing it. I'll let hem provide more details, but my understanding is:

1. Yes
2. Compression support isn't implemented yet, but it sounds like you'd find mixed methods desirable? I think @jorgehermo9 planned to work on that next.

[jorgehermo9]

Yes! If the UDP datagram does not contain the gelf magic bytes, the bytes will be passed "as is", so "unchunked" messages will also work.

Decompression and its automatic detection is on work by me. It will be transparent for the user, though. I'm planning to submit a PR these weeks.

Thanks for your feedback @arch-xtof'. I really appreciate it and it is very motivating for me :)

[arch-xtof]

That's great to hear. While automatic decompression will be useful, for now, this logic can be easily implemented with VRL. I’m looking forward to trying this out. Thanks so much for the hard work in bringing this feature.

[jorgehermo9]

Hi @arch-xtof. Just opened #21816 to address the chunked decompression support.

Feel free to build it from the branch and test if it works for your usecase. Feedback is very welcome

[jorgehermo9]

We also had the exact same problem.
as the framing is independent of the used codec, the workaround we used was

[sources.test]
type = "socket"
mode = "udp"
address = "0.0.0.0:4242"
decoding.codec = "json"
framing.method = "chunked_gelf"

So the gelf codec does not validate the json payload that it's receiving

I think that should work in your case

[arch-xtof]

Thanks, I figured that out and also did the decompression part in VRL.

    [sources.logger_gelf]
      type = "socket"
      address = "${POD_IP}:12201"
      mode = "udp"
      framing.method = "chunked_gelf"

    [transforms.logger_decompressed]
      type = "remap"
      inputs = ["logger_gelf"]
      source = '''
        magic_bytes, err = encode_base16(slice!(.message, start: 0, end: 2))
        if err != null { abort }

        if magic_bytes == "789c" {
            .message, err = decode_zlib(.message)
        } else if magic_bytes == "1f8b" {
            .message, err = decode_gzip(.message)
        }

        if err != null {
          log(err, level: "warn", rate_limit_secs: 60)
          abort
        }
      '''

[jorgehermo9]

I bet that the decompression in your case should be automatically detected and done inside the chunked_gelf framer. Can you try without decompressing it in VRL?

Also, if your messages are at least json-compliant, remember that you can use this

    [sources.logger_gelf]
      type = "socket"
      address = "${POD_IP}:12201"
      mode = "udp"
      decoding.codec = "json"
      framing.method = "chunked_gelf"

[arch-xtof]

Ooh, I misunderstood it then. I thought decompression would only work if I used decoding.codec = "gelf", but now I understand that decompression is inside of framing.method = "chunked_gelf". In this case, everything works perfectly. I tried it out, great design.

[jorgehermo9]

I tried it out, great design.

Thank you very much! Thats very heartwarming!!