filelog: How to declare multiline-configuration based on pod-information (e.g. annotations)?

[ChristianCiach]

A common pattern to collect all kubernetes-logs on a node looks like this:

receivers:
  filelog/pods:
    config:
      include:
        - /var/log/pods/*/*/*.log
      include_file_name: false
      include_file_path: true
      operators:
        - type: container

Of course, the log-files retrieved this way will contain wildly different log-formats (jsonl, glog, klog, spring-boot, ...) and each log-format may have different multiline rules.

Most log-shippers/collectors that I am familiar with allow the recombination of log-events later in the pipeline, when the meta-information of the pods is already known. In Otelcol, the pod details are usually added by the k8sattributes processor, which happens sometime after the filelog-receiver has emitted its data.

But unfortunately, there doesn't seem to be a way to combine log-events as a processor? It looks like I have to use a recombine operator as part of the filelog configuration, but there is no way to access the pod details (like pod.annotations["log-format"]) at this stage to make an informed decision about the multiline configuration.

Since there is no way to infer the log-format based on a single log-event, the "best" I could to is to hardcode multiline rules based on the information of the container operator, like namespace, pod-name and container-name. But this forces me (the cluster-operator) to hardcode log processing for specific workloads that I may not (and don't want to) even know about.

Surely there has to be a better, more flexible way to do that in Otelcol?

[ChristianCiach]

A solution I came up with:

receivers:
  receiver_creator/logs:
    watch_observers: [k8s_observer]
    receivers:
      filelog/pods:
        rule: >-
          type == "pod.container"
        config:
          include:
            - /var/log/pods/`pod.namespace`_`pod.name`_`pod.uid`/`container_name`/*.log
          include_file_name: false
          include_file_path: true
          operators:
            - type: container
              add_metadata_from_filepath: false  # Metadata already added by receiver_creator
            - type: router
              routes:
                - expr: >-
                    '`pod.annotations["log-format"]`' == 'spring-boot'
                  output: merge_springboot
                - expr: >-
                    '`pod.annotations["log-format"]`' == 'my-company-python-flavour'
                  output: merge_python
              default: after_recombine
            - id: merge_python
              type: recombine
              combine_field: body
              is_first_entry: '...'
              overwrite_with: oldest
              output: after_recombine
            - id: merge_springboot
              type: recombine
              combine_field: body
              is_first_entry: '...'
              overwrite_with: oldest
              output: after_recombine
            - type: noop
              id: after_recombine

This uses the receiver_creator for the sole purpose of having pod-details available at filelog-level and to allow expr-evaluation inside the filelog config. This way, we should be able to construct a router-operator that routes to recombine-operators specific to pod-annotations. I didn't test it, but it should work somehow like that.

This is fine, but I don't love it. I also wonder about the performance and resource-consumption when creating separate filelog receivers for each container vs. creating a single filelog receiver for all log files.