Internal_metrics add additional context from Kubernetes

[szibis]

Is there any plan to add additional more universal approach in enrich context of internal_metrics ??.

Just like pid_key or host_key It would be great to add specific service from Kubernetes or any other dimension to all component scopes and not only use static defined tags added to all internal_metrics.

[jszwedko]

@szibis do you mean to add tags to the internal metrics based on Vector's Kubernetes metadata? That is: when Vector is running in k8s, pull its own metadata and add it to the internal metrics?

[szibis]

Like not on Kubernetes only but in this case yes.
In general, internal_metrics can be enriched by additional tags but these tags can be static or dynamic but only at the host/node level where Vector is operating. We can add things like env, project, team, and more but no more than that.
What will really help in debugging things in multi-tenancy environments is to ability to map for example specific labels/annotations of pods/containers and enrich internal_metrics on each level by using this data just like in normal flow kubernetes_logs enrich based on the pod, namespace, or node metadata.
Something similar was possible in Fluentd internal metrics.
It would be great to find context for example from Kubernetes service/namespace in internal metrics for all levels of transformation next to component tags that already Vector exposes - this could be just a list of labels/annotations to include in enriching.
I see here a great thing in operational finding problems for specific services or specific parts of Vector topology by alerting specific Teams that this part is owned for example.
Any prefixing or suffixing for each part of the topology to find the place where this is happening is not a very good idea.
Analyzing the logs is always painful and having that kind of insight on the Dashboard from metrics could help in Vector OPEX.

[jszwedko]

Gotcha, yeah, I see. This would amount to tagging internal telemetry that is directly related to events (like component_sent_events_total) with information from the events itself. We have some plans in this area to do this with specific tags (service and source_id) specifically for event processing metrics (components_(received|sent)_events_total and component_(received|sent)_event_bytes_total) in #17322 but we do see a future where the tags could be dynamic.

[szibis]

Thanks for the information. Even this #17322 looks better than current setup.