tag_cardinality_limit per metric

[NasAmin]

Hi,

We are starting to use tag_cardinality_limit transform to enforce some cardinality controls on our tags.
Vector is setup to receive metrics from dozens of applications. Vector seems to be printing the cardinality metric like this:

{
  "name": "value_limit_reached_total",
  "namespace": "vector",
  "tags": {
    "component_id": "drop_high_cardinality_tags",
    "component_kind": "transform",
    "component_name": "drop_high_cardinality_tags",
    "component_type": "tag_cardinality_limit"
  },
  "timestamp": "2022-11-15T14:32:51.657818181Z",
  "kind": "absolute",
  "counter": {
    "value": 1
  }
},
{
  "name": "tag_value_limit_exceeded_total",
  "namespace": "vector",
  "tags": {
    "component_id": "drop_high_cardinality_tags",
    "component_kind": "transform",
    "component_name": "drop_high_cardinality_tags",
    "component_type": "tag_cardinality_limit"
  },
  "timestamp": "2022-11-15T14:32:51.657818181Z",
  "kind": "absolute",
  "counter": {
    "value": 1080
  }
}

Looking at the internal logs, I am getting entries like this:

{
  "timestamp": "2022-11-15T14:31:46.054273Z",
  "level": "DEBUG",
  "message": "Rejecting tag after hitting configured value_limit.",
  "tag_key": "name",
  "tag_value": "7bfadae4ec1fc4eb7247fedd385641314940085384283b7b45df5d4f6e8a51a1",
  "internal_log_rate_secs": 10,
  "target": "vector::internal_events::tag_cardinality_limit",
  "span": {
    "component_id": "drop_high_cardinality_tags",
    "component_kind": "transform",
    "component_name": "drop_high_cardinality_tags",
    "component_type": "tag_cardinality_limit",
    "name": "transform"
  },
  "spans": [
    {
      "component_id": "drop_high_cardinality_tags",
      "component_kind": "transform",
      "component_name": "drop_high_cardinality_tags",
      "component_type": "tag_cardinality_limit",
      "name": "transform"
    }
  ]
}

As you can see, the above log entry just says tag_key: name which is the most common tag name. How will I know which metric its coming from?

This implies that Vector is just counting cardinality for a label regardless of which metric its coming from. Is this supposed to be for tag cardinality per metric or is it independent of the metric?

If its the later then it doesn't really seem to be useful. I would appreciate you help in understanding this transform and its correct usage.

What I am really looking for is how will I determine which metric is emitting a high cardinality tag and then talk to the relevant team to fix it.

Kind regards,

Nas

[NasAmin]

See #15395