query became very slow in k8s (milvus 2.5.3)

[cherrylzhao]

hello everyone.
I have deployed Milvus-cluster 2.5.3 in k8s with helm, today a simple query took about 20 seconds, I have found some WARN in query node like this

./my-release-milvus-querynode-b9c5ff97f-w7jrv.log:[2025/02/28 12:15:37.991 +00:00] [WARN] [querynodev2/handlers.go:227] ["failed to query on delegator"] [traceID=268e6c2161234e885622cc89eb222369] [msgID=456325833448226818] [collectionID=455327742181110652] [channel=by-dev-rootcoord-dml_3_455327742181110652v0] [scope=All] [error="context deadline exceeded"]
./my-release-milvus-querynode-b9c5ff97f-w7jrv.log:[2025/02/28 12:16:20.069 +00:00] [WARN] [delegator/delegator.go:527] ["delegator query failed to wait tsafe"] [traceID=33bd58f7bc43bc2ab4d8cdedac387535] [collectionID=455327742293841420] [channel=by-dev-rootcoord-dml_6_455327742293841420v0] [replicaID=455327778458305210] [error="context deadline exceeded"]
./my-release-milvus-querynode-b9c5ff97f-w7jrv.log:[2025/02/28 12:16:20.069 +00:00] [WARN] [querynodev2/handlers.go:227] ["failed to query on delegator"] [traceID=33bd58f7bc43bc2ab4d8cdedac387535] [msgID=456325844484489217] [collectionID=455327742293841420] [channel=by-dev-rootcoord-dml_6_455327742293841420v0] [scope=All] [error="context deadline exceeded"]
./my-release-milvus-querynode-b9c5ff97f-w7jrv.log:[2025/02/28 12:17:15.722 +00:00] [WARN] [delegator/delegator.go:527] ["delegator query failed to wait tsafe"] [traceID=3c7eac97388d20091270bc0315c55cb6] [collectionID=455327742180136058] [channel=by-dev-rootcoord-dml_11_455327742180136058v0] [replicaID=456232816274833462] [error="context deadline exceeded"]
./my-release-milvus-querynode-b9c5ff97f-w7jrv.log:[2025/02/28 12:17:15.723 +00:00] [WARN] [querynodev2/handlers.go:227] ["failed to query on delegator"] [traceID=3c7eac97388d20091270bc0315c55cb6] [msgID=456325859072802817] [collectionID=455327742180136058] [channel=by-dev-rootcoord-dml_11_455327742180136058v0] [scope=All] [error="context deadline exceeded"]

all other nodes are ok, query node resource usage as below

NAME                                           CPU(cores)   MEMORY(bytes)
my-release-milvus-querynode-b9c5ff97f-bwq6j    134m         42959Mi         
my-release-milvus-querynode-b9c5ff97f-fsmxm    69m          40894Mi         
my-release-milvus-querynode-b9c5ff97f-kzjhv    171m         39881Mi         
my-release-milvus-querynode-b9c5ff97f-w7jrv    137m         40721Mi 

I have 4 woker node with 32c x 128G, and query node was scheduled on 4 nodes, it seems that cpu & memory resource is ok,

and I also find the queryNode.scheduler.maxReadConcurrentRatio parameters can improve the perf, and how can I upgrade the setting using helm & verify it.

please give me some hints for that, thanks !

[yhmo]

1. Deploy a monitoring service to observe milvus cluster: https://milvus.io/docs/monitor.md
   Observe the status of query nodes, including "Collection Loaded Num", "Segments Loaded Num", "Time Tick Behind Now", "Search Request Latency", "Search in Queue Latency", "Flowgraph Num".
   

2. Try search(consistency_level="Eventually")

[cherrylzhao]

thanks，consistency_level="Eventually" can resolve the problem, we have get some hints from the issue #34392, so we install the ntpd service in k8s to avoid the time of pod is not synchronized.

by the way, how can we configure dashboard as you do above, is there any guides for that, thanks!

[yhmo]

Configure the dashboard: https://milvus.io/docs/visualize.md

[cherrylzhao]

I have imported the milvus-dashboard.json, but no data is displayed.

I can see default metrics in k8s pod resource in grafana, how can I revise the settings, thanks!

[yhmo]

I guess you didn't follow the steps in this doc: https://milvus.io/docs/monitor.md

[cherrylzhao]

metric data can display successful, thanks!