Elasticsearch Sink Stats

[tastyfrankfurt]

Hey Team,

Been fiddling with the elasticsearch sink over last couple of days, having an issue with the stats reporting incorrectly. Basically the scenario is as follows,

1. Bulk Post 3000 events
2. 1 of those events fails to map to a template correctly so elasticsearch rejects it but accepts the other 29999 events
3. Vector reports dropping 3000 events from internal logging
4. Vector top reports error on the sink and the stats show that 3000 less events went in

Hopefully that makes sense, basically it gets to a point where vector top reports the sink as say 2m events in and on the out queue it reports 1m events out.

Not sure if someone else can confirm this behavior

Thanks

Tasty

[jszwedko]

Hi @tastyfrankfurt ,

Unfortunately that is the current behavior. #140 is tracking improving this to only retry failed records.

[tastyfrankfurt]

Thanks for confirming @jszwedko much appreciated, was a bit worried we were dropping alot more.