filter processor include rather than drop

[yuzhouliu9]

In older versions of the filter processor there was the ability to define the behavior as include or exclude

For example, only include logs that match the regex

  filter/regexp:
    logs:
      include:
        match_type: regexp
        resource_attributes:
          - key: host.name
            value: prefix.*

versus exclude logs that match the regex

  filter/regexp:
    logs:
      exclude:
        ...

However this has been deprecated and no longer available. The current filter processor only drops logs on match.
Sometimes writing the OpenTelemetry Transformation Language condition to exclude all logs EXCEPT a few particular conditions is difficult. It would be simpler to write the include logic.

Is it possible to achieve the include behavior like in older filter processors?

Thanks!

[edmocosta]

AFAIK, the old configuration style is still available (at least for now). It was removed from the docs (#31722) to prioritize the OTTL configuration style, but it should still be working with the latest collector versions.

The OTTL style should also be simple to use, for example, your config:

  filter/regexp:
    logs:
      include:
        match_type: regexp
        resource_attributes:
          - key: host.name
            value: prefix.*

Can be translated to the OTTL condition IsMatch(resource.attributes["host.name"], "prefix.*"), then all you need to do is to leverage the not operator to make it similar to the include or exclude behavior.

Include:

filter/regexp:
  logs:
    log_record:
      - not IsMatch(resource.attributes["host.name"], "prefix.*")

exclude:

filter/regexp:
  logs:
    log_record:
      - IsMatch(resource.attributes["host.name"], "prefix.*")

Here you can see the old style vs new style in action for the include use-case.

Multiple conditions are evaluated using the or operator, but if that makes it confusing, you can always combine all conditions into a single statement, for example:

logs:
  log_record:
      - not(
          HasPrefix(resource.attributes["host.name"], "prefix.") or
          HasPrefix(resource.attributes["host.name"], "another.")
        )

Hope that helps :)