Increase expiration period for "log_to_metric"/"prometheus_exporter"

[arch-xtof]

Hello!

I am trying to count incoming log messages based on specific fields. For example, if I'll receive the following entry:
{"message": "blabla", "facility": "some-service", "level": "error"}
Vector will transform it into the following Prometheus counter metric:
vector_sent_log_entries_count{facility="some-service",level="error"} 1

Version
0.30.0 running as Deployment in k8s

Problem
I have no expire_metrics global parameter configured. Despite this, when no new log entries are received for over 5 minutes, metrics expire. Can I somehow increase the expiration period?

Relevant Vector Config

    [transforms.count_logs]
      type = "log_to_metric"
      inputs = [ "filter_logs" ]

      [[transforms.count_logs.metrics]]
        type = "counter"
        field = "facility"
        name = "sent_log_entries_count"
        namespace = "vector"
        increment_by_value = false

        tags.kind = "logger"
        tags.env = "test"
        tags.region = "eu-central-1"
        tags.region_domain = "eu-central-1..."
        tags.facility = "{{ facility }}"
        tags.level = "{{ level }}"

    [sinks.prom_metrics]
      type = "prometheus_exporter"
      inputs = ["count_logs"]
      address = "${POD_IP}:12201"
      default_namespace = "vector"

[jszwedko]

HI @arch-xtof ,

Not sure if this is your issue, but there is a separate expiration of metrics in the promotheus_exporter sink that you can configure: https://vector.dev/docs/reference/configuration/sinks/prometheus_exporter/#flush_period_secs . It defaults to 60s though so the fact that you are seeing it happen after 300s makes me think it might not be related. Though maybe Grafana is doing some interpolation 🤔 Maybe try bumping that expiration and see what happens.

[arch-xtof]

Hello @jszwedko,

First of all, that you for your suggestion. I tried it initially and nothing changed, so I thought this parameter was not what I needed. Today I used curl to check the metrics and discovered that metrics had a timestamp. Prometheus by default does not scrape metrics created longer than 5 minutes ago. I used the suppress_timestamp = true parameter to fix my issue.

[jszwedko]

Aha, that makes sense. Glad you were able to resolve it!