How Do I Modify the Vector Sink Configuration to Reduce the Error Reported in ES 429?

[sunyoona]

vector.yaml:

sinks:
  k8s_log:
    type: elasticsearch
    endpoints:
      - https://elk-elasticsearch-client-headless.elk.svc.cluster.local:9200
    inputs:
      - k8s_log
    auth:
      strategy: basic
      user: "xxxxx"
      password: "xxxxx"
    batch:
      timeout_secs: 3600
      max_events: 100000
    buffer:
      max_size: 8388608000
      type: disk
      when_full: block
    compression: gzip
    request:
      rate_limit_duration_secs: 1
      rate_limit_num: 10000
    bulk:
      index: "dev-{{ .log_source }}-%Y.%m.%d"
    tls:
      ca_file: "/usr/share/vector/certs/ca.crt"

When the amount of logs increases, I get a lot of errors:

WARN sink{component_kind="sink" component_id=k8s_log component_type=elasticsearch}:request{request_id=388}: vector::sinks::util::retries: Retrying after response. reason=too many requests internal_log_rate_limit=true

ERROR sink{component_kind="sink" component_id=k8s_log component_type=elasticsearch}:request{request_id=389}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_429" response=Response { status: 429, version: HTTP/1.1, headers: {"x-elastic-product": "Elasticsearch", "content-type": "application/json; charset=UTF-8", "content-length": "565"}, body: b"{\"error\":{\"root_cause\":[{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=318706156, replica_bytes=0, all_bytes=318706156, coordinating_operation_bytes=4010923, max_coordinating_and_primary_bytes=322122547]\"}],\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of coordinating operation [coordinating_and_primary_bytes=318706156, replica_bytes=0, all_bytes=318706156, coordinating_operation_bytes=4010923, max_coordinating_and_primary_bytes=322122547]\"},\"status\":429}" }

Some of the options I've configured don't seem to solve this error, if there are better vector configuration parameters, or modify the configuration of ES.

[pront]

Root Cause:

• The Elasticsearch coordinating node queue is full — specifically the coordinating_and_primary_bytes.
• Your coordinating node has a max limit of ~322 MB (max_coordinating_and_primary_bytes=322122547 bytes). Right now, it's at ~318 MB used — almost completely full — so ES rejects your request with HTTP 429.

You can try with:

    batch:
      timeout_secs: 180
      max_events: 50000