Elasticsearch Sink with data_stream not working

[minimax75]

Hello,
I try to setup an Elastic sink with a data stream, but Elastic is not accepting my data. I alway receive following error:

Error-Log:
2022-11-02T14:30:54.037527Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-11-02T14:30:54.037634Z INFO vector::app: Loading configs. paths=["/etc/vector"]
2022-11-02T14:30:54.528763Z INFO vector::topology::running: Running healthchecks.
2022-11-02T14:30:54.529440Z INFO vector: Vector has started. debug="false" version="0.24.2" arch="x86_64" build_id="5e47b89 2022-10-13"
2022-11-02T14:30:54.529840Z INFO source{component_kind="source" component_id=fluent component_type=fluent component_name=fluent}: vector::sources::util::tcp: Listening. addr=0.0.0.0:24224
2022-11-02T14:30:54.530139Z INFO source{component_kind="source" component_id=vector component_type=vector component_name=vector}: vector::sources::util::grpc: Building gRPC server. address=0.0.0.0:6000
2022-11-02T14:30:54.530280Z INFO vector::internal_events::api: API server running. address=127.0.0.1:8686 playground=off
2022-11-02T14:30:54.530453Z INFO source{component_kind="source" component_id=syslog component_type=syslog component_name=syslog}: vector::sources::util::tcp: Listening. addr=0.0.0.0:9000
2022-11-02T14:30:54.552088Z INFO vector::topology::builder: Healthcheck: Passed.
2022-11-02T14:30:54.574614Z INFO vector::topology::builder: Healthcheck: Passed.
2022-11-02T14:31:37.450137Z ERROR sink{component_kind="sink" component_id=elk-shared-dev component_type=elasticsearch component_name=elk-shared-dev}:request{request_id=1}: vector::internal_events::elasticsearch: Response failed. error_code=http_response_400 error_type="request_failed" stage="sending" response=Response { status: 400, version: HTTP/1.1, headers: {"x-elastic-product": "Elasticsearch", "content-type": "application/json;charset=utf-8", "content-length": "261"}, body: b"{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}" }
2022-11-02T14:31:37.450239Z ERROR sink{component_kind="sink" component_id=elk-shared-dev component_type=elasticsearch component_name=elk-shared-dev}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}"
......

Vector is running with role "Aggregator" and installed via helm.

my config for the sink:
elk-shared-dev:
type: elasticsearch
inputs:
- sbox
auth:
user: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
strategy: basic
bulk:
action: create
index: vector-test
endpoint: https://elasticsearch:9200
mode: data_stream
compression: none
healthcheck:
enabled: true

I already tried several configs, but still not working.
Any hint?

Many Thanks

[spencergilbert]

👋

It sounds like you're using a version of ES that doesn't support _type so you should be using: https://vector.dev/docs/reference/configuration/sinks/elasticsearch/#suppress_type_name

[minimax75]

Hi,
I'm running a Elastic 8.3.3.

[spencergilbert]

Yes, _type is not supported in that version, so you should use the linked issue to suppress that.

[minimax75]

Hi,

many thanks! Works now!

here my working config for Elastic 8.3:
elk-shared-dev:
type: elasticsearch
inputs:
- sbox
auth:
user: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
strategy: basic
endpoint: https://elasticsearch:9200
mode: bulk
bulk:
action: index
index: vector-test
suppress_type_name: true
compression: none
healthcheck:
enabled: true

[spencergilbert]

I'll also note that we'll be adding a feature that will try to dynamically determine the version and ensure a compatible config for you, that should be available with the next stable release.