PubSub does not support metric events?

[genadipost]

I am trying to use Vector agent to collect metrics from a local Prometheus server as to forward the data to Vector aggregator with GCP PubSub as a relay.
Also i am trying to avoid using metric_to_log and log_to_metric transforms.
https://vector.dev/highlights/2022-03-31-native-event-codecs/

With the following architecture in mind:

Prometheus <-- Vector Agent -> GCP PubSub <-- Vector Aggregator (Prometheus exporter sink) <-- Prometheus
(left to right)

1. Vector agent scrapes Prometheus server
2. Vector agent publishes the metrics to GCP PubSub
3. Vector Aggregator reads metrics from PubSub
4. Vector Aggregator sinks data via Prometheus exporter
5. Prometheus server scrapes Vector Aggregator

This is the configuration of the agent.

customConfig:
  data_dir: /vector-data-dir
  api:
    enabled: true
    address: 0.0.0.0:8686
  sources:
    host_metrics:
      type: host_metrics
    prometheus_kube:
      type: prometheus_scrape
      endpoints:
        - http://prometheus-kube-prometheus-prometheus.kube-prometheus.svc.cluster.local:9090/metrics
  sinks:
    pub_sub:
      type: gcp_pubsub
      project: abc
      topic: my-topic
      encoding:
        codec: native
      inputs:
      - prometheus_kube

I am getting the following error:

2024-09-18T21:20:34.048629Z  INFO vector::app: Log level is enabled. level="info"
2024-09-18T21:20:34.050070Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2024-09-18T21:20:34.050807Z ERROR vector::cli: Configuration error. error=Data type mismatch between prometheus_kube (Metric) and pub_sub (Log)

It seems GCP PubSub does not support metric events.

Should i resort to metric_to_log and log_to_metric?

[genadipost]

Found a discussion in discord.
Metrics in PubSub are not supported.

https://discord.com/channels/742820443487993987/1131992466241900707

[julienlau]

for clarity :

sjoeboo — 7/21/23, 6:53 PM
Just started poking around with Vector, and my use case seems fairly simple: prometheus remote_write for a source, GCP pubsub as a sink. But it seems like (while i can't find anything in the sink docs), the PubSub sink only supports logs? But the native encoding/native_json encoding both support metrics? Have I just missed something to do i really need to convert the metrics to logs (then back again on the otherside?)

jszwedko — 7/21/23, 10:24 PM

Hi @sjoeboo ! Yeah, it looks like the pubsub sink only accepts logs at the moment. It might just need removing the & DataType::Log from

vector/src/sinks/gcp/pubsub.rs

Line 147 in 0f14c0d

Input::new(self.encoding.config().input_type() & DataType::Log)

. We didn't do that originally for all sinks, when the codecs were introduced, because some sinks make assumptions about the data types that need to be loosened. Giving the GCP PubSub sink a quick scan it doesn't look like one of them so we could probably just remove that requirement and add a test. Feel free to open a PR if you are motivated

[pront]

We didn't do that originally for all sinks, when the codecs were introduced, because some sinks make assumptions about the data types that need to be loosened. Giving the GCP PubSub sink a quick scan it doesn't look like one of them so we could probably just remove that requirement and add a test. Feel free to open a PR if you are motivated

Right, this is the case for many components that impose restriction on data types.

[julienlau]

I guess the fact that pub-sub does not preserve order is not compatible with out-of-order samples ingestion on metrics storage.
Kafka only preserves message order on a single partition not globally or on a topic.