[Bug]: Elasticsearch: All shards failed

[LewisRye]

What happened?

I have installed Elasticsearch as a Jaeger backend through the helm chart, and whenever I run a load on my deployment, traces are not collected and Jaeger UI returns error 503: service unavailable (all shards failed). It should be noted that everything works as intended, when there are less traces.

I believe this error is caused by lack of memory allocated to Elasticsearch, and I can verify this with the low amount of memory allocated through kubectl. However, I do not know how to increase the limit. I have tried the Java heap allocation environment variable, and kubernetes resource requests, but to no success.

Does anyone know why this is happening?

Steps to reproduce

1. Load Elasticsearch through helm chart
2. Run a load to create traces
3. Keep running load
4. Try to view traces through Elasticsearch API or Jaeger UI
5. Error 503 all shards failed

Expected behavior

I want to give Elasticsearch more memory so it doesn't error 503.

Relevant log output

Screenshot

Additional context

No response

Jaeger backend version

No response

SDK

No response

Pipeline

No response

Storage backend

Elasticsearch

Operating system

No response

Deployment model

No response

Deployment configs

[dosubot[bot]]

@LewisRye Hello there! I'm here to help you with your Elasticsearch issue. Let's get to the bottom of this together.

To address the "all shards failed" error in Elasticsearch when used as a Jaeger backend, you can try the following steps to increase the memory allocation for Elasticsearch:

1. Increase Memory Resources: Adjust the memory resources allocated to Elasticsearch in your Helm chart values. You can do this by modifying the resources section in your values.yaml file for the Elasticsearch deployment. For example:

   resources:
     requests:
       memory: "4Gi"
     limits:
       memory: "8Gi"

   This configuration requests 4Gi of memory and sets a limit of 8Gi for Elasticsearch.

2. Adjust Elasticsearch Heap Size: Ensure that the Elasticsearch heap size is appropriately set. You can set the heap size by configuring the ES_JAVA_OPTS environment variable. For example:

   env:
     - name: ES_JAVA_OPTS
       value: "-Xms4g -Xmx4g"

   This sets the minimum and maximum heap size to 4Gi.

3. Check Elasticsearch Configuration: Verify other Elasticsearch configurations that might affect performance, such as the number of shards and replicas. You can adjust these settings based on your load requirements.

4. Monitor Performance: Use monitoring tools to observe Elasticsearch's performance and resource usage. This can help identify if the issue is indeed related to memory or if there are other bottlenecks.

These steps should help mitigate the "all shards failed" error by ensuring Elasticsearch has sufficient resources to handle the load.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}