How to set resource attribute "service.name" from log record attribute?

[BrudiBanani]

Hi,
I have kind of similar question like already asked in #35883. But there are some key differences so I opened a new discussion.

I am not able to set the resource attribute "service.name" as it seems to me that the resource attribute processor is not able to grab attributes from log records.

I have following (shortend) otel-collector config

receivers:
  filelog:
    include_file_path: true
    include:
      - /var/lib/docker/containers/**/*log
    operators:
      - id: container-parser
        type: container
        format: docker
        add_metadata_from_filepath: false

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 50
    spike_limit_percentage: 30
  batch:
  transform:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
        - set(attributes["my.service"], attributes["attrs"]["tag"])
  attributes/container_name:
    actions:
      - key: attrs
        action: delete
  resource/service_name:
    attributes:
      - key: service.name
        action: insert
        from_attribute: my.service

exporters:
  otlphttp/loki:
    endpoint: http://{{ monitoring['loki']['hostname'] }}:{{ monitoring['loki']['ports']['http'] }}/otlp
    tls:
      insecure: true

extensions:
  health_check:

service:
  pipelines:
      receivers: [filelog]
      processors: [transform, attributes/container_name, resource/service_name, batch]
      exporters: [otlphttp/loki]

And here is a sample log line:

{"log":"LogRecord #56\n","stream":"stderr","attrs":{"tag":"otel-collector"},"time":"2024-11-16T11:31:40.531701198Z"}

And some debug output from the transform processor:

"TransformContext": {
        "resource": {
            "attributes": {},
            "dropped_attribute_count": 0
        },
        "scope": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "",
            "version": ""
        },
        "log_record": {
            "attributes": {
                "log.iostream": "stderr",
                "attrs": {
                    "tag": "otel-collector"
                },
                "log.file.name": "6e6d5eb5-json.log",
                "log.file.path": "/var/lib/docker/containers/6e6d5eb5-json.log",
                "my.service": "otel-collector"
            },
            "body": "2024-11-16T11:24:27.840+0100\tdebug\tottl@v0.113.0/parser.go:338\tTransformContext after statement execution",
            "dropped_attribute_count": 0,
            "flags": 0,
            "observed_time_unix_nano": 1731752667945056439,
            "severity_number": 0,
            "severity_text": "",
            "span_id": "0000000000000000",
            "time_unix_nano": 1731752667841114950,
            "trace_id": "00000000000000000000000000000000"
        },
        "cache": {}
    }

So the transform processor is working as intended i guess? Atleast it finds the attributes["attrs"]["tag"] and creates my.service correctly.
However the resource processor is not able to pick it up. How can I achieve that?

Many thanks.

[BrudiBanani]

Using the transform processor kind of works. The following configuration does set the service name.
However it does not work properly.

Somewhere in the processing of the log files the wrong service.names are set. I end up getting the wrong log lines attached to I am also not sure how to debug this correctly.
Example Log:

time=2024-12-22T09:33:27+01:00 attrs_tag=abcd service_name=wxyz msg=Example Message

The filelog receiver is configured to be able to look into every log file of any container. As I have multple containers running the value in "attributes["attrs"]["tag"]" differs depending on from which container it is set.
I would expect that the transform processor is processing every incoming log line individually. But that does not seem to be the case as I am ending up with the wrong service name on a lot of log lines.
Has anyone an idea how to configure that correctly?

receivers:
  filelog:
    include_file_path: true
    include:
      - /var/lib/docker/containers/**/*log
    operators:
      - id: container-parser
        type: container
        format: docker
        add_metadata_from_filepath: false

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 50
    spike_limit_percentage: 30
  batch:
  transform:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
        - set(resource.attributes["service.name"], attributes["attrs"]["tag"])

exporters:
  otlphttp/loki:
    endpoint: http://loki.example.com/otlp

service:
  telemetry:
    metrics:
      level: normal
    logs:
      level: INFO
  pipelines:
    logs:
      receivers: [otlp, filelog]
      processors: [transform, batch]
      exporters: [otlphttp/loki]

Many thanks!

[edmocosta]

It seems your logs lines are being grouped into the same resource, which explain why the transformation is having this unexpected behaviour. I think enabling the transform.flatten.logs should help.

[BrudiBanani]

Thanks for the hint! Works like a charm :)

[kubebn]

Hi, I’m trying to achieve the same result, but service_name is still being set as unknown_service.

receivers:
  filelog:
    include_file_path: true
    include:
      - /var/lib/docker/containers/**/*log
    operators:
      - id: container-parser
        type: container
        format: docker
        add_metadata_from_filepath: false

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 50
    spike_limit_percentage: 30
  batch:
  transform:
    error_mode: ignore
    flatten_data: true
    log_statements:
      - context: log
        statements:
          - set(resource.attributes["service.name"], attributes["container_name"])

exporters:
  otlphttp/loki:
    endpoint: "${OTLP_ENDPOINT}"
    headers:
      "X-Scope-OrgID": "${OTLP_ORG_ID}"
    auth:
      authenticator: basicauth/loki

extensions:
  basicauth/loki:
    client_auth:
      username: "${OTLP_USERNAME}"
      password: "${OTLP_PASSWORD}"

  health_check:
    endpoint: 0.0.0.0:13133
    path: /health

  pprof:
    endpoint: 0.0.0.0:1777

service:
  extensions: [health_check, pprof, basicauth/loki]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform, batch]
      exporters: [otlphttp/loki]

any advice on this?

[edmocosta]

It seems to be a different issue. I'm not pretty sure from where the unkown_service came from, but I think the attribute you're copying the value from attributes["container_name"] is probably nil, and AFAIK, that metadata is not automatically added when reading it from docker log paths (not k8s).

That said, you probably don't actually need the transform processor for setting the service.name, and maybe something like this would work for you. Please note I haven't tested it, but it might be useful as a starting point :)

  filelog:
    include_file_path: true
    include:
      - /var/lib/docker/containers/**/*log
    operators:
      - id: container-parser
        type: container
        format: docker
        add_metadata_from_filepath: false
      - id: extract-container-name
        type: regex_parser
        regex: '.*\/containers/(?P<container_name>[^/]+)\/.*'
        parse_from: attributes["log.file.path"]
      - id: move-container-name-to-service-name
        type: move
        from: attributes.container_name
        to: resource["service.name"]