Working configuration for kubernetes logs with elasticsearch sink

[YuryHrytsuk]

I am using helm chart v0.43.0 (i.e. vector v0.47.0). I cannot manage to make vector work as I keep facing object mapping for [kubernetes.pod_labels.app] tried to parse field [app] as object, but found a concrete value issue

Coud someone point out what I am missing here to deal with ES object mapping issue? Also documenting this clearly will help (this is a basic use case I presume)

# values.yaml for Vector Helm chart
# https://github.com/vectordotdev/helm-charts/blob/vector-0.43.0/charts/vector/values.yaml


# Useful example
# https://github.com/vectordotdev/vector/issues/19136#issuecomment-1809927271


role: "Agent"

# avoid error
#
#  Release "vector" does not exist. Installing it now.
#  Error: 1 error occurred:
#      * Service "vector" is invalid: spec.ports: Required val
service: 
  enabled: false

env:
  - name: ELASTICSEARCH_USERNAME
    value: &elasticsearch_username elastic
  - name: ELASTICSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elasticsearch-es-elastic-user
        key: *elasticsearch_username

customConfig:
  # https://github.com/vectordotdev/helm-charts/issues/226
  data_dir: /vector-data-dir

  sources:
    kubernetes_logs:
      type: kubernetes_logs

  transforms:
    # https://github.com/vectordotdev/vector/issues/3588
    dedot_labels_for_es:
      type: remap
      inputs:
        - kubernetes_logs
      source: |-
        . = map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }

  sinks:
    # https://vector.dev/docs/reference/configuration/sinks/elasticsearch
    es_out:
      type: elasticsearch
      inputs: ["dedot_labels_for_es"]
      endpoint: "http://elasticsearch-es-http.{{.Release.Namespace}}.svc:9200"
      auth:
        strategy: basic
        user: "${ELASTICSEARCH_USERNAME}"
        password: "${ELASTICSEARCH_PASSWORD}"
      bulk:
        action: index
        index: "vector-k8s-%Y-%m-%d"

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 200m
    memory: 256Mi

Related issues:

• [kubernetes_logs] Logs are not forwarded if pod has "app" label configured  #8058
• object mapping for [kubernetes.labels.app] tried to parse field [app] as object, but found a concrete value #19136
• Handle dynamic field mapping with the dot notation in the elasticsearch sink #3588

[pront]

Hi @YuryHrytsuk, it is highly likely you are hitting this issue: #8058 (comment).

Can you try this workaround to confirm it's the same issue?