Events miss/ Data loss

[yahya-zuberi]

Hello Team,
So we were using mix panel to capture our mobile events and get insights of user behaviour from it. We came with a case that we need this data in our system as our analyst team needs to do some insights of that data and make querying and availability easy. For this case we came across vector.dev and found it suitable for our scenario.

So our vector works in a way that our mobile sdk send events to vector and then vector sinks it to mixpanel as well as to s3 buckets. Vector is running in a Kubernetes environment on AWS eks using c6i.2xlarge(as recommended on documentation) node with ALB attached on it to handle and load balance traffic.

We did a small POC and things looked fine. All the data was captured and processed as expected.
When taken into production with 100% load the issue we started to face is events getting missed. We noticed a significant drop in our events being captured. The mobile side is verified that all the events are being published but not sure what can be the case on vector side.

Looking forward for suggestions and discussion to figure out the root cause. One hypothesis is that ALB is causing a bottleneck when events are fired to vector on high throughput.

[jszwedko]

Hi @yahya-zuberi !

Could you share the Vector configuration and version you are using? I have a couple of thoughts, but seeing the configuration would help focus the discussion.