High cardinality metrics in k8s environments

[ArtemTrofimushkin]

Hello to everyone!

Some vector metrics have potentially unbound cardinality, like following:

• vector_files_added_total
• vector_files_unwatched_total
• vector_files_resumed_total
• vector_processed_bytes_total

Because each metrics contains label "file" (autogenerated by log driver) or "pod_name" (autogenertated by k8s).
I tried to mitigate these issues with configuration transforms:

sources:
  kubernetes_logs:
    type: kubernetes_logs
  internal_metrics:
    type: internal_metrics
transforms:
  prometheus_sink_transform_cleanup:
    inputs:
    - internal_metrics
    source: |
      del(.tags.file)
      del(.tags.pod_name)
    type: remap
  prometheus_sink_transform_aggregate:
    inputs:
    - prometheus_sink_transform_cleanup
    type: aggregate
sinks:
  prometheus_sink:
    address: 0.0.0.0:9090
      inputs:
      - prometheus_sink_transform_aggregate
      type: prometheus_exporter

Here first remap transform is used to delete high cardinality tags and then aggregate transform is used to aggregate multiple metric events values to a single metric event.
But it seems, that after the first transform, the metric values began to be calculated incorrectly - displayed only the first value, not the sum of all metrics events. For example:

(before transform)

# HELP vector_files_resumed_total files_resumed_total
# TYPE vector_files_resumed_total counter
vector_files_resumed_total{component_kind="source",component_name="kubernetes_logs",component_type="kubernetes_logs",file="/var/log/pods/dev-mlc_cookinginfo-create-queue-cws5p_ea730cd7-731d-4865-b871-c65977c80659/create-queue/0.log"} 1 1628443140171
vector_files_resumed_total{component_kind="source",component_name="kubernetes_logs",component_type="kubernetes_logs",file="/var/log/pods/dev-cc_datacatalog-migrator-zgw6j_3d14844c-68da-4d78-9401-418a0d448b9d/migrator/0.log"} 1 1628443140171
vector_files_resumed_total{component_kind="source",component_name="kubernetes_logs",component_type="kubernetes_logs",file="/var/log/pods/dev-ninja_ratings-migrator-qgdxc_df6be0da-9211-4490-bda0-2ddfbbe9fadf/migrator/0.log"} 1 1628443140171
vector_files_resumed_total{component_kind="source",component_name="kubernetes_logs",component_type="kubernetes_logs",file="/var/log/pods/dev-mct_shopwindow-migrator-mk2tm_bac2c59e-36e5-4c0c-94fa-8f4e8ab5e3a7/migrator/0.log"} 1 1628443140171
vector_files_resumed_total{component_kind="source",component_name="kubernetes_logs",component_type="kubernetes_logs",file="/var/log/pods/dev-ninja_ratings-56847b6d64-lggl6_5d3d5cc8-666e-4deb-b5e4-0832112bcd75/backend/0.log"} 1 1628443140171
...

(after transform):

# HELP vector_files_resumed_total files_resumed_total
# TYPE vector_files_resumed_total counter
vector_files_resumed_total{component_kind="source",component_name="kubernetes_logs",component_type="kubernetes_logs"} 1 1628443267424

I also read about tag cardinality limit. But it seems, this transform is more like a safeguard for high cardinality metrics.

Is there any way to drop high cardinality tags and aggregate metric values?

[spencergilbert]

Hey! Is this a preventative step or are you seeing cardinality issue while running today. I would have expected a counter being run through aggregate to sum the metrics, I'll see if I can check the code and verify that's the intended behavior for counter metrics.

[ArtemTrofimushkin]

Hello!
Yes, it's about cardinality issue that we observe in prometheus tsdb status for a while (at TSDB Status -> Head Cardinality Stats)

[spencergilbert]

It certainly seems like this configuration should work - I'll see if I can reproduce with a simple scenario and track down the bug

[jszwedko]

The issue here appears to be that files_resumed_total is reported as an absolute counter rather than an incremental one. I'm looking a little closer.

[jszwedko]

Yeah, it seems like internal_metrics reports all counters as absolute since it aggregates them.

I think the solution here is #3668 to allow more types of aggregations. I added a note there that we should ensure that we are able to aggregate across metric tags. We are planning to pick that work up within the next couple of months.

[ArtemTrofimushkin]

Thanks for the explanation!

[wtchangdm]

Hi all, just wanted to report I am also seeing high cardinality on vector_files_unwatched_total, and google lead me to this discussion.

I will try to add the transform that @ArtemTrofimushkin is using. @jszwedko @spencergilbert Do you think this situation deserves a few lines in documents?