SW_STORAGE_ES_RESPONSE_TIMEOUT not work in k8s

[xiaozaiyuji]

I helm install the skywalking oap server 10.1.0, after for a while,the query is slow and data increase,so I modify the environment SW_STORAGE_ES_RESPONSE_TIMEOUT 60000,but the oap server print the error

2025-07-02T17:33:13.646875020+08:00 2025-07-02 09:33:13,646 org.apache.skywalking.library.elasticsearch.ElasticSearch 156 [armeria-eventloop-epoll-9-4] WARN [] - [creqId=8acd5778, preqId=2ecca77a, chanId=1acd7c2a, laddr=10.201.202.190:38002, raddr=elasticsearch-master-headless.elasticsearch.svc.cluster.local/10.201.101.139:9200][http://elasticsearch-master-headless.elasticsearch.svc.cluster.local:9200/sw_metrics-all/_search#GET] Response: {startTime=2025-07-02T09:33:13.645Z(1751448793645516), length=0B, duration=0ns, totalDuration=15000ms(15000449600ns), cause=com.linecorp.armeria.client.ResponseTimeoutException, headers=[:status=0]}

2025-07-02T17:33:13.646881985+08:00 com.linecorp.armeria.client.ResponseTimeoutException: null

2025-07-02T17:33:13.646887255+08:00 at com.linecorp.armeria.client.ResponseTimeoutException.get(ResponseTimeoutException.java:38) ~[armeria-1.29.3.jar:?]

2025-07-02T17:33:13.646893372+08:00 at com.linecorp.armeria.internal.common.DefaultCancellationScheduler.invokeTask(DefaultCancellationScheduler.java:514) ~[armeria-1.29.3.jar:?]

2025-07-02T17:33:13.646898817+08:00 at com.linecorp.armeria.internal.common.DefaultCancellationScheduler.lambda$start$2(DefaultCancellationScheduler.java:148) ~[armeria-1.29.3.jar:?]

2025-07-02T17:33:13.646903435+08:00 at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45) [armeria-1.29.3.jar:?]

2025-07-02T17:33:13.646907857+08:00 at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646912489+08:00 at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646917124+08:00 at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646930226+08:00 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646934961+08:00 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646939544+08:00 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) [netty-transport-classes-epoll-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646943927+08:00 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646948374+08:00 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646952815+08:00 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.108.Final.jar:4.1.108.Final]

2025-07-02T17:33:13.646957561+08:00 at java.lang.Thread.run(Unknown Source) [?:?]

I check the environment is effective,so this is the bug of skywalking?
I config the application.yml by the configmap,and change the value to 60000,this log print as always,I don't think this is a config problem,this is an bug for skywalking.

[wu-sheng]

Why do you think a network time out is client error?
This is usually network or DNS issue.
Client doesn't have capabilities to control that. And demo was running in k8s and elastic.

[xiaozaiyuji]

This problem to timeout is so boring and I can not to increase the es resource memory or cpu.

[wu-sheng]

As I replied, I don't have that experience about using a so slow elastic.

[xiaozaiyuji]

I tested the performance for elasticsearch with kibana，to my surprise，this is not very slow。This search only use 500ms，the elasticsearch is not slow.

But when I search the es just with trace_id this is so slow. I think there is something wrong.

I did may traceId query with the kibana,there is some slow query for different traceId. But the result will show in kibana,but no traceId serach result will show in skywalking ui,this is a big question no matter how.

[xiaozaiyuji]

I think this is not a question about elasticsearch this is a bug of skywalking.

[wu-sheng]

I don't have any idea, as it can't be reproduced. Everything works well on my side, the slow thing is only showing up at your end.
You could easily say it is a bug of skywalking, but that doesn't help anyone actually.
There is no magic we could tell how things went slow.

[xiaozaiyuji]

I did not mean this,I downgrade to 9.7.0 this parameter effect, the search can up to 30 second without timeout.So I think in 10.1.0 this is a bug,the parameter is not effect.