Vector agent adds/unwatches the same container log file multiple times

[svenmueller]

Hi everyone! Is it expected for a Vector agent (running in K8s) to find the same Kubernetes container log file and stop watching it over and over (we are running latest version 0.18.0)? We currently have a lot of missing Kubernetes container log events and therefore wonder if this behavior is normal.

[jdrouet]

Hi @svenmueller!
Did you run a previous version before? Could you share with me the configuration you're using for this source?
Thanks 😉

[svenmueller]

Hi @jdrouet,

Thx for the response! See my answers below.

Did you run a previous version before?

Yes, we used different versions for the agents. Here is the full history in correct order: 0.17.3 -> 0.15.2 -> nightly-2021-11-18 -> 0.18.0 (current version)

Could you share with me the configuration you're using for this source?

Sure! Below is the configuration we use for Vector agents.

customConfig:
  data_dir: "/vector-data-dir"
  sources:
    kubernetes:
      type: kubernetes_logs
      max_line_bytes: 131072
      delay_deletion_ms: 3600000 
    internal_logs:
      type: internal_logs
    internal_metrics:
      type: internal_metrics

  transforms:
    parse_internal_logs:
      type: remap
      inputs:
        - internal_logs
      source: |-
        .cloud_provider.service = "vector"
        .host = "${HOSTNAME}"
  sinks:
    vector-aggregator:
      type: vector
      inputs:
        - kubernetes
        - parse_internal_logs
      address: vector-aggregator-haproxy.logging.svc:9080
      healthcheck:
        enabled: true
    prometheus-exporter:
      type: prometheus_exporter
      inputs:
        - internal_metrics
      address: 0.0.0.0:9090

Thanks a lot

[spencergilbert]

My initial assumption here is the log file is being rotated out by Kubernetes - and when we start watching the new file it still has the same name.

[svenmueller]

I think your assumption is correct 👍