Got too many parse json error when we facing to huge volume logs

[syzcch]

Hi Team,

The log format:

{..................
}
EOF
---------------
{................
}
EOF
---------------

Vector version 0.23.3
Our config:

data_dir = "/local/vectorLog"
    [sources.gz_logs]
    type = "file"
    max_line_bytes = 1024000
    ignore_older_secs = 600
    include = [ "/.../year/month/**/request*.gz" ]
    multiline.mode = "halt_with"
    multiline.condition_pattern = "-----------------------------------------------------------------------"
    multiline.timeout_ms = 1000
    multiline.start_pattern = '^\{'
    # Parse gz_logs
    [transforms.parse_logs]
    type = "remap"
    inputs = ["gz_logs"]
    source = """
    lines,err = split(.message, "\n")
    .message = lines[0] + lines[1]
    .message = to_string(.message)
    .message = parse_json!(.message)
    """
    [sinks.print]
    type = "kafka"
    inputs = [ "parse_logs" ]
    bootstrap_servers = "ip"
    topic = "topic1"
    compression = "none"
     [sinks.print.encoding]
      codec = "json"

Got too many error messages when Vector facing huge volume of logs. The error like the following messages.

2022-09-28T18:26:50.846129Z ERROR transform{component_kind="transform" component_id=parse_logs component_type=remap component_name=parse_logs}: vector::internal_events::remap: Internal log [Mapping failed with event.] has been rate limited 3959 times.
2022-09-28T18:26:50.846175Z ERROR transform{component_kind="transform" component_id=parse_logs component_type=remap component_name=parse_logs}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"parse_json\" at (122:143): unable to parse json: EOF while parsing an object at line 1 column 23465" error_type="conversion_failed" stage="processing" internal_log_rate_secs=10
2022-09-28T18:26:50.846977Z ERROR transform{component_kind="transform" component_id=parse_logs component_type=remap component_name=parse_logs}: vector::internal_events::remap: Internal log [Mapping failed with event.] is being rate limited.

I don't know why it happened. But I found that there are lots of logs missing the last two characters "}" due to the errors, so we can not parse it via Logstash due to invalid json format.
Do I need to adjust something? like max_line_bytes or multiline.timeout_ms to fix it?
Thank you so much.

[neuronull]

Hi @syzcch ,

Apologies for the delayed response!

Is this an issue you are still facing?

It looks like something in your remap transform might be off. The parse_json! is seeing an EOF in the middle of parsing an object.

Could you provide a sample log that is outputted from the file source (becoming the input to the parse_logs transform.

You could acquire such a sample by creating a console sink (or file sink) that takes the gz_logs as input.
Knowing what the transform is receiving as input will help us determine whether there is an issue in configuring the gz_logs source , or the parse_logs transform.

[syzcch]

Hi @neuronull ,

Thank you for your reply. As for this issue, I guess that not all log items follow the same pattern so we need to find out the special log patterns firstly. We are still working on that.

Another issue seems to be too hard to solve. Please check the discussion here: #14592 . Could you please help us on this issue? Thank you so much.