Source vector under the hood

[bmallya]

Hello,
What does "vector" when used as a sink and source combination use under the hood? I am assuming it is some sort of a tcp connection?
And is there a provision to say sink to multiple sources using some sort of multicast.

Thanks.

[jszwedko]

The vector source / sink use gRPC to send data over HTTP/2. There isn't any multicast support (I don't think). Typically we've recommended users put a load balancer in between the sink and source Vector instances.

[bmallya]

Thanks @jszwedko. The reason I was asking this was that I am trying to do the following
Client A | | Collector X
Client B | ---(all clients forward events using vector sink)------Load Balancer (ha proxy) -----------> | Collector Y
Client C | | Collector Z
With the following entry in the haproxy cfg
listen logService bind *:50336 mode http server name1 172.10.19.32:50335 check port 8008 server name2 172.10.19.33:50335 check port 8008

I am noticing the following message on the client (and the events are not being forwarded). The version that I am using right now is v0.22.0
'2023-06-22T08:33:40.652242Z WARN sink{component_kind="sink" component_id=forward component_type=vector component_name=forward}:request{request_id=0}: vector::sinks::util::retries: Retrying after error. error=Request failed: status: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 503 Service Unavailable", details: [], metadata: MetadataMap { headers: {"content-length": "107", "cache-control": "no-cache", "content-type": "text/html"} }
'

[jszwedko]

Hmm, we actually use HAProxy as an example load balancer in our Helm chart. You might find the config there useful: https://github.com/vectordotdev/helm-charts/tree/develop/charts/vector/templates/haproxy

Otherwise you might find https://www.haproxy.com/documentation/hapee/latest/load-balancing/protocols/grpc/ useful for configuring HAProxy to forward gRPC traffic.

[bmallya]

Thanks @jszwedko , let me take a look.

[k24dizzle]

@bmallya did you end up finding a fix? I am running into the same error trying to get logs to go from vector sink -> haproxy -> vector source

[bmallya]

@k24dizzle , yes I was able to get it work. This was my haproxy configuration
listen logService mode http bind *:50336 proto h2 default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions server server1 172.17.10.132:50335 proto h2 check server server2 172.17.10.133:50335 proto h2 check

The vector sink was straight forward
[sinks.forward] type = "vector" inputs = ["input1"] address = "ha-proxy:50336" version = "2"