Decompression at Source/Remap (GZIP)

[arch-xtof]

Hello!

I know that vector sinks can compress the data before sending it out. Is there any way that sources or remaps can decompress data before they send it to the next stage?

In my case, I have GZIP compressed data available via UDP which I would like to process via Vector. Unfortunately, I have no option of removing compression.

[jszwedko]

Hey @arch-xtof !

Unfortunately I think your best option currently would be to use Lua to do the decompression. We will likely add functions to do this via remap though. E.g. zlib. Would you want to open a similar issue for gzip?

[arch-xtof]

Hello @jszwedko !

Thank you for letting me know about the Lua option.

I looked deeper into my problem and noticed that logs (formatted in JSON) have absolutely no delimiters. So even if I manage to decompress them by Lua, they won't be delimited.

This feature will not solve my current problem, but it sounds like a good addition to Vector. I'll be happy to open an issue.