Vector Source kubernetes_log Using Pod Annotations

[cahyahikmawan]

Vector Agent version: v0.23.0
Installed by: Helm

Vector Configuration

data_dir: /vector-data-dir
sources:
  kubernetes_logs:
    extra_field_selector: metadata.annotations.logging/enabled=true
    type: kubernetes_logs
sinks:
  kafka:
    bootstrap_servers: server:port
    encoding:
      codec: json
    inputs:
      - kubernetes_logs
    topic: topic_logs
    type: kafka

Output Log

2022-08-02T05:25:00.234216Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "field label not supported: metadata.annotations.logging/enabled", reason: "BadRequest", code: 400 }))

Pod Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx
  replicas: 2
  template:
    metadata:
      annotations:
        logging/enabled: "true"
      labels:
        app.kubernetes.io/name: nginx
        logging/enabled: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

I have tried filter using extra_label_selector:logging/enabled=true and it works fine, but how it work using annotations.
I've been looking for documentation, issues, discussions, but can't find how to do it.
Thank you!

[tobz]

As it turns out, this is a limitation in the Kubernetes API itself (source). Even the official Kubernetes docs for the watch endpoint don't list out the allowed fields, and it's not clear to me if the allowed fields are even protected in terms of API stability.

I would suggest to keep using the label selector-based approach if that works for you.