Using docker container envrioment variables as loki log labels

[dpewsey]

Looking to lift environment variables out of docker and use to define loki log labels. Wondering if anyone knows if this is possible.
Here is the config toml

          [api]
            enabled = true
            address = "0.0.0.0:8686"
            playground = true
          [sources.logs]
            type = "docker_logs"
          [sinks.out]
            type = "console"
            inputs = [ "logs" ]
            encoding.codec = "json"
          [sinks.loki]
            type = "loki"
            inputs = ["logs"]
            endpoint = "url"
            encoding.codec = "json"
            healthcheck.enabled = true
            [sinks.loki.labels]
              testing = '{{ env.\\[1\\] }}'`

I've tried with this and escaping out the characters.

Anyone any ideas?

Thanks

[spencergilbert]

Hi @dpewsey - to clarify, are the env variables in the container that Vector is running in - or are you looking for variables from the containers the logs were collected from?

[sources.logs]
  type = "docker_logs"
[sinks.loki]
  type = "loki"
  inputs = ["logs"]
  endpoint = "http://url:3100"
  encoding.codec = "json"
  healthcheck.enabled = true
  [sinks.loki.labels]
    testing = '"${ENV}'

The above would populate the testing label with an env var VAR, set in Vector's container (ref).

The second part (getting variables from the container that generated the log) probably isn't possible.

[dpewsey]

Hi @spencergilbert
Ah yeah it would be to for the container that generated the log.

I saw someone doing this for nomad, as was hoping I could use the same concept.

  type = "loki"
  inputs = ["logs"]
  endpoint = "http://loki.example"
  encoding.codec = "json"
  healthcheck.enabled = true
  # since . is used by Vector to denote a parent-child relationship, and Nomad's Docker labels contain ".",
  # we need to escape them twice, once for TOML, once for Vector
  labels.job = "{{ label.com\\.hashicorp\\.nomad\\.job_name }}"
  labels.task = "{{ label.com\\.hashicorp\\.nomad\\.task_name }}"
  labels.group = "{{ label.com\\.hashicorp\\.nomad\\.task_group_name }}"
  labels.namespace = "{{ label.com\\.hashicorp\\.nomad\\.namespace }}"
  labels.node = "{{ label.com\\.hashicorp\\.nomad\\.node_name }}"

[spencergilbert]

If it's a label on the container, we should be able to access it - could you share an example log that you've gotten out of a console sink.

[dpewsey]

that was just the example I saw and can get working. It was hoping could use the same concept to get the ENV from the container.

Here a log example in loki, it works great however on the loki end can only use the parser to sort them and it doesn't count a proper "log label" shows as a "detected field"

  | {   "container_created_at": "2022-07-18T12:04:46.357045358Z",   "container_id": "583e8f34e1e01353fe23c81777d748e4a1df09ac82811f32ed20c869839dd0ae",   "container_name": "server-80070f20-4603-c718-1686-33b7f2fc4b21",   "host": "31a713ff4142",   "image": "627432529318.dkr.ecr.eu-west-1.amazonaws.com/altmetric-explorer:build-9162",   "label": {     "test_label": "server-pod",     "service": "test-service"   },   "message": "I, [2022-07-18T13:25:25.617098 #8]  INFO -- : [1be9a182-e0b6-4e98-b38d-3497ed0efa40] Started GET \"/ping\" for 1.9.168.0.1 at 2022-07-18 13:25:25 +0000",   "source_type": "docker",   "stream": "stdout" }
-- | --

The only option to quest by is agent=vector or if I hard set something else at [sinks.loki.labels]

[spencergilbert]

I suspect being able to access the env's of other containers may constitute a security risk.

Maybe something like:

[sinks.loki.labels]
label_*: "{{ label }}"

That should get you targetable tags on your Loki logs like label_test_label and label_service (based on your example).

[dpewsey]

I think that'll be our best bet, was just hoping we could extract preexisting things from the Envs and save myself a job.

Thanks for you help!