How to drop a log record if it cannot be parsed?

[gajus]

I've configured a parser using Kubernetes labels:

fluentbit.io/parser: airbyte

The parser itself looks like so:

[PARSER]
    name airbyte
    format regex
    -- https://regex101.com/r/lSdtR8/1
    regex ^(?<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})\s(?<severity>[^ ]*)\s(?<namespace>[^ ]*)\s-\s(?<message>.*)$

My problem is that the target pods are producing logs in a variety of formats that do not conform with the parser, many of which I do not care for. Those logs effective skip the parser.

Is there a way for me to entirely trop those logs if they do not match the parser?

At the moment, I need to write a filter for every possible log format they produce, e.g.

[FILTER]
    name grep
    match *
    # These are low value/high noise (LVHN) logs produced by airbyte workers
    # https://regex101.com/r/IIFK6X/1
    exclude log /^Log4j2Appender says:/
    # following by many more exclude rules...

which feels like a losing battle

[FBnil]

If it does not match the PARSER, you need to intercept it with a FILTER. In this case, if it does not parse, it will not have a "namespace", so we just grab anything in that field with ".".
If it DID parse, and you want to ignore certain other messages, use the Logical_Op in "or" mode to write less filters.

[INPUT]
    Name   dummy
#    Dummy {"time": "2023-12-31 12:13:14", "severity": "critical", "namespace":"space", "message":"should be parsed"}
    Dummy {"time": "123456", "log": "Log4j2Appender says: Records read: 36009000 (15 GB)"}
    Samples 1
    tag unclassified

[FILTER]
    Name grep
    Match unclassified
    Regex namespace .

[FILTER]
    name grep
    match *
    # These are low value/high noise (LVHN) logs produced by airbyte workers
    # https://regex101.com/r/IIFK6X/1
    Logical_Op or
    exclude message /^Log4j2Appender says:/ 
    exclude message /Records read/
    
[OUTPUT]
    Name   stdout
    Match  *

For more examples, please read the documentation: https://docs.fluentbit.io/manual/pipeline/filters/grep

That being said, if you are using Fluentbit 3 or higher, there are other ways to exclude messages, which might be better for you: metrics in processors have an action exclude. And expect filters have an action exit. You might want to look into those too.

https://isitobservable.io/observability/kubernetes/exploring-fluent-bit-v3-http-2-support-new-processors-and-more
https://docs.fluentbit.io/manual/local-testing/validating-your-data-and-structure