How metrics behave with Vector with multiple pods

[bmgante]

Hi All,

I am using vector helm chart to receive http json events (source), apply logs to metrics transformations and expose the metrics in prometheus endpoint (sink).

I´ve been testing it with 1 replica (default) and it seems to work properly, but i am quite confused on how vector balances events if i have replica set to 2 if i need to scale up the service.

# replicas -- Specify the number of Pods to create. Valid for the "Aggregator" and "Stateless-Aggregator" roles.
replicas: 2

Can someone provide some guidance on this?

Thanks

[jszwedko]

Hi @bmgante ! The typical recommendation is to set up a load balancer to receive the traffic and route to the Vector instances. There is some discussion around this here: https://vector.dev/docs/setup/going-to-prod/sizing/#horizontal-scaling

[bmgante]

Hi @jszwedko i have ingress enabled (nginx) on vector helm chart which points to the vector service.

Service:

vector                                        ClusterIP   a.b.c.d    <none>        8686/TCP,9090/TCP,6000/TCP   7d5h
vector-headless                               ClusterIP   None             <none>        8686/TCP,9090/TCP,6000/TCP   7d5h

The deployment method is statefulset which can have multiple replicas/pods. I just would like to know how metrics creation is handled by these multiple pods, i guess it´s transparent and its managed internally but i am curious on how events are balanced between the pods to guarantee metrics are properly created/updated.

[jszwedko]

I'm not sure I follow the question exactly, but each Vector instance is independent and has no shared state with other instances. It's up to a load balancer in-front of Vector to do the balancing.

[bmgante]

maybe... my sink is a prometheus endpoint, and there is only one prometheus endpoint no matter of how many pods/replicas i have configured. I was under the impression there was some shared state between the pods but i can be wrong and it´s just simpler than i thought.

[jszwedko]

Gotcha, yeah, each replica is completely independent so scraping the prometheus exporter endpoint of one should return the metrics that only that Vector instance has seen.