Jaeger - Distributed Tracing Platform
Jaeger ingester with OpenSearch #5139
Unanswered
gfes980615
asked this question in
Q&A
Replies: 1 comment
-
|
All there parameters are pass-through to Elasticsearch, I suggest reading documentation there (or in the olivere ES driver) |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi team
Issue
I used the jaeger ingester with OpenSearch
Sometimes I will receive the 429 error, the error message like the following
{ "level": "error", "ts": 1705999951.4621732, "caller": "config/config.go:146", "msg": "Elasticsearch could not process bulk request", "request_count": 732, "failed_count": 2, "error": "elastic: uncommitted bulk respo nse items", "response": { "took": 9, "errors": true, "items": [ { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "xTOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488356, "_pri mary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "xjOFNY0BGQIQjPT5ZkRb", "status": 429, "error": { "type": "circuit_breaking_exception", "reason": "[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16809625030/15.6gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16809617624/15.6gb], new bytes reserved: [7406/7.2kb], usages [request=0/0b, fielddata=3437877477/3.2gb, in_flight_requests=7406/7.2k b]" } } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "xzOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489162, "_primary_term": 1, "status": 201 } }, { "index": { "_index": " jaeger-span-2024-01-23-08", "_id": "yDOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442930, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "yTOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489163, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "yjOFNY0BGQIQjPT5ZkRb ", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442931, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "yzOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "cr eated", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442932, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zDOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442933, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zTOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488357, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zjOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7487800, "_primar y_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zzOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442934, "_primary_term": 1, "status": 201 } }, { " index": { "_index": "jaeger-span-2024-01-23-08", "_id": "0DOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488358, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-sp an-2024-01-23-08", "_id": "0TOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7465703, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "0j OFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7487801, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "0zOFNY0BGQIQjPT5ZkRb", "status ": 429, "error": { "type": "circuit_breaking_exception", "reason": "[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16809625030/15.6gb], which is larger than the limit of [16320875724/15.1gb], real usage: [1680961762 4/15.6gb], new bytes reserved: [7406/7.2kb], usages [request=0/0b, fielddata=3437877477/3.2gb, in_flight_requests=7406/7.2kb]" } } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1DOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "creat ed", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488359, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1TOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "s uccessful": 2, "failed": 0 }, "_seq_no": 7465704, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1jOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_ seq_no": 7442935, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1zOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489164, "_primary_t erm": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "2DOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488360, "_primary_term": 1, "status": 201 } }, { "ind ex": { "_index": "jaeger-span-2024-01-23-08", "_id": "2TOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489165, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span- 2024-01-23-08", "_id": "2jOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442936, "_primary_term": 1, "status": 201 } } ] }, "stacktrace": "github.com/jaegertracing/jaeger/pkg/es/conf ig.NewClient.func2\n\tgithub.com/jaegertracing/jaeger/pkg/es/config/config.go:146\ngithub.com/olivere/elastic.(*bulkWorker).commit\n\tgithub.com/olivere/elastic@v6.2.37+incompatible/bulk_processor.go:588\ngithub.com/olivere/elastic.(* bulkWorker).work\n\tgithub.com/olivere/elastic@v6.2.37+incompatible/bulk_processor.go:487" }Question
I suppose the message means the bulk request bytes are too large, I need to tune the bulk config with Jaeger ingester
And the configs I need to tune are
--es.bulk.actions,--es.bulk.size, and--es.bulk.workersIs it right?
Does bulk request byte will be the
--es.bulk.actions*--es.bulk.size*--es.bulk.workers?Will any other config affect the bulk request bytes?
Or does it have nothing to do with these settings?
Beta Was this translation helpful? Give feedback.
All reactions