buffer messages when network is bad and sync the messages to remote when network is good

[janiu-001]

Hi expert,

I read the vector doc, it is really a good tool to sync data from source to dest, and the source and sinks support more than 40 types.

Would you help on a question?

1. our service are deployed on the customer server, it data could be stored in kafka/rabbitmq
2. the destination is a http server

From the guide, it is possible to sync the data from kafka/rabbitmq to http server, but in my case, the network is not good always, our customer only could connect to internet sometimes, could it support by vector? could it work with acknowledgements.enabled?

Very appreciated for you response

[janiu-001]

For the http sinks, the url is like below, could support multiple uri? and could the url be dynamic like: https://10.22.212.22:9000/:clientId/:deviceId

[jszwedko]

This is not currently supported. See #1155

[janiu-001]

@jszwedko @neuronull would you help on above questions?

[jszwedko]

Hi @janiu-001 ,

Yes, Vector will buffer events if the downstream sink is unavailable. See https://vector.dev/docs/about/under-the-hood/architecture/buffering-model/#buffering-at-the-sink for details about this. Enabling acknowledgements will have Vector wait to ack the messages in Kafka/RabbitMQ until the sink has accepted them (or they've been persisted to a disk buffer). Otherwise Vector will ack the messages as soon as it receives them.

Hope this helps! Let us know if you have more questions.