Telemetry metrics

[erpiterora]

Hi there, I am new to vector and still struggling with syntaxes etc. I am trying to figure out how does one go about using the telemetry metrics (component_received_event_bytes_total, component_received_events_count etc...).

Could someone please post a simple example?

Thanks in advance!

[jszwedko]

Hi @arpitarora2022 ! Typically users will use an internal_metrics source with a metrics sink to publish the metrics to an external service (like Prometheus).

[erpiterora]

Thanks @jszwedko, that helped. I am running into a slightly different issue now. Here's my TOML file.
How do I set the host field for the Splunk HEC metrics?

For some reason, the host field is being set to an IP address instead of 'test_host' I am trying to set it to.

[sources.my_internal_metrics]
type = "internal_metrics"
scrape_interval_secs = ${METRICS_SCRAPE_INTERVAL_SECS}
namespace = "${METRICS_NAMESPACE}"

[transforms.get_sink_metrics_only]
type = "filter"
inputs = ["my_internal_metrics"]
condition = '.tags.component_id == "${METRICS_SOURCE_COMPONENT_ID}"'

[transforms.add_host_field]
type = "remap"
inputs = ["get_sink_metrics_only"]
source = '''
        .ingest_host = "test_host"
        .host = "test_host"
'''

[sinks.splunk_hec_sink]
type = "splunk_hec_metrics"
inputs = ["add_host_field"]
endpoint = "${SPLUNK_HEC_ENDPOINT}"
index = "${METRICS_INDEX}"
source = "${METRICS_SOURCE}"
compression = "none"
token = "${METRICS_HEC_TOKEN}"
sourcetype = "${METRICS_SOURCETYPE}"

[erpiterora]

I was able to figure it out. Here's the code. You basically set the tags.host_key to 'host' in the internal_metrics source and then you add the remap transform and set the host there. In this example, I am changing the host to 'test_host'.

[sources.my_internal_metrics]
type = "internal_metrics"
scrape_interval_secs = ${METRICS_SCRAPE_INTERVAL_SECS}
namespace = "${METRICS_NAMESPACE}"
tags.host_key = "host"

[transforms.set_component_id_filter]
type = "filter"
inputs = ["my_internal_metrics"]
condition = '.tags.component_id == "${METRICS_SOURCE_COMPONENT_ID}"'

[transforms.vector_metrics_1]
type = "remap"
inputs = ["set_component_id_filter"]
source = '''
        .tags.host = "test_host"
'''

[sinks.splunk_hec_sink]
type = "splunk_hec_metrics"
inputs = ["vector_metrics_1"]
endpoint = "${SPLUNK_HEC_ENDPOINT}"
index = "${METRICS_INDEX}"
source = "${METRICS_SOURCE}"
compression = "none"
token = "${METRICS_HEC_TOKEN}"
sourcetype = "${METRICS_SOURCETYPE}"

[erpiterora]

I have one more question!

How do I add custom tags to an 'internal_metrics' source?

Do I need to convert the internal metrics to logs using 'metric_to_log', remap and add tags there and then convert the logs to metrics using 'log_to_metric'?

This sounds a bit convoluted. Is there any easier way where I can add custom tags into the 'internal_metrics' output? So far, I see it only allows for the 'host_key' and 'pid_key' tags. I want to add the 'environment' tag to the 'internal_metrics' output.

[jszwedko]

Hi @arpitarora2022 ! You can use a remap transform to add tags as you have in #15291 (comment) . No need to convert to a log event first. E.g. .tags.foo = "bar".

[erpiterora]

Thanks, that worked!!! Here's the latest code in case it helps anyone in the future.
Vector rocks!!!

[sources.my_internal_metrics]
type = "internal_metrics"
scrape_interval_secs = ${METRICS_SCRAPE_INTERVAL_SECS}
namespace = "${METRICS_NAMESPACE}"
tags.host_key = "host"

[transforms.set_component_id_filter]
type = "filter"
inputs = ["my_internal_metrics"]
condition = '.tags.component_id == "${METRICS_SOURCE_COMPONENT_ID}"'

[transforms.vector_metrics_1]
type = "remap"
inputs = ["set_component_id_filter"]
source = '''
        .tags.host = "test_host"
        .tags.environment = "${METRICS_ENVIRONMENT}"
'''

[sinks.splunk_hec_sink]
type = "splunk_hec_metrics"
inputs = ["vector_metrics_1"]
endpoint = "${SPLUNK_HEC_ENDPOINT}"
index = "${METRICS_INDEX}"
source = "${METRICS_SOURCE}"
compression = "none"
token = "${METRICS_HEC_TOKEN}"
sourcetype = "${METRICS_SOURCETYPE}"