Elasticsearch sink issue

[LinTechSo]

Hi.
i set up kafka ------> vector[es sink] ------> es
after 11:40 in screenshot my data insertion in Elasticsearch became zero in some cases and i don't know why. after i restarting my vector dev on 18:30 pm the issue seems to be fixed.

i am using vector using 0.26.0 with kafka source and elasticsearch sink

would you please help me figure this out
i never have seen this before in previous versions

[LinTechSo]

this is my vector configuration

  data_store:
    type: elasticsearch
    inputs:
      - to_json
    compression: none
    endpoints: 
      - "https://test.te"
    api_version: auto
    mode: bulk
    bulk:
      action: index
      index: "test-%F"
    healthcheck:
      enabled: true
    auth:
      user: "test"
      password: "test"
      strategy: "basic"
    acknowledgements:
      enabled: true

[LinTechSo]

unfortunately i didn't checkout the error logs. i check the vector metrics but finding nothing special

[jszwedko]

Gotcha. I'd recommend checking the error logs next time it happens.

[LinTechSo]

Hi @jszwedko . it happened again.
error logs:

2023-01-03T07:49:13.728618Z ERROR source{component_kind="source" component_id=input_logs component_type=kafka component_name=input_logs}: rdkafka::client: librdkafka: Global error: OperationTimedOut (Local: Timed out): kafka-0.kafka-headless.default.svc.cluster.local:9092/1: 1 request(s) timed out: disconnect (after 61061ms in state UP, 1 identical error(s) suppressed)    
2023-01-03T07:50:15.396556Z ERROR source{component_kind="source" component_id=input_logs component_type=kafka component_name=input_logs}: rdkafka::client: librdkafka: Global error: OperationTimedOut (Local: Timed out): kafka-0.kafka-headless.default.svc.cluster.local:9092/1: 1 request(s) timed out: disconnect (after 61666ms in state UP, 1 identical error(s) suppressed) 

and warning logs:

2023-01-03T10:00:46.592257Z  WARN sink{component_kind="sink" component_id=data_store component_type=elasticsearch component_name=data_store}:request{request_id=1}: vector::sinks::util::retries: Request timed out. If this happens often while the events are actually reaching their destination, try decreasing `batch.max_bytes` and/or using `compression` if applicable. Alternatively `request.timeout_secs` can be increased. internal_log_rate_limit=true

i set up kafka and vector and es inside the k8s cluster and the connections are internally in a k8s cluster with svc.

i disabled health check in sink now and i think it would help. but i cant find out why this heppened.

[jszwedko]

Hmm, that's odd. Could it be that the Elasticsearch instance isn't responding within the timeout?

[LinTechSo]

thanks for reply.
yes could be. but now i disabled health check for a couple of days and this problem didn't happen again.