How to handle NULL Byte Padded Log format (Meta Trader 4 Server / Trading Platform)?

[tamer-hassan]

Someone already explained this in their best effort here: https://discuss.elastic.co/t/how-to-read-logs-with-permanent-null-characters-at-the-end-of-file/324408

To sum it up briefly, Meta Trader 4 Server, a trading platform by MetaQuotes, writes latest log files in quite a peculiar way:
It pads the file with some Megabytes of NUL Bytes (\x000) in one long line, and upon writing some log lines / messages, it replaces some NUL bytes, and appends the remaining NUL bytes to the end/last line of the log.

This effectively confuses the file source in vector, and I could not find a way around it. Vector follows some, but then stops seeing new log lines / messages altogether at some point.

What I've tried:

1. attempted to filter the line beginning with NUL byte before remap:

[sources.mt4_main_logs]
type = "file"
include = ["D:\\MT4Server\\logs\\*.log"]]

[transforms.mt4_main_logs_filter]
type = "filter"
inputs = [ "mt4_main_logs" ]
condition = """
    starts_with(string!(.message), "\\0") == false
"""

result: warnings are still printed to stdout frequently by vector about discarding line over max allowed bytes (102400?), and vector stops seeing new messages / log lines after some minutes (effectively stopping sending to sink the latest log lines). So the problem is still at the file source.

2. Tried setting ignore_checkpoints = true to the file source

result: same as above. warnings about discarding of line above max allowed bytes, and vector stops seeing new log messages after some minutes, and has to be restarted to re-read the current log and start sending to the sink what it has missed , from where it left off.

Any ideas or input would be much appreciated!

Cheers