Impact to Searches when the Query Node is brought down and brought up again.

[xiaobingxia-at]

Hey team, I have a question regarding bringing down and then bring up the query node. When I drain the EC2 node at EKS cluster, causing the query node brought down, the query node will release its segments through balancing process, and at the same time the rest of query nodes will load these segments.

Is this migration process able to guarantee that: during the whole process all searches towards these segments being migrated are not impacted?

Based on our observation, when the query node doesn't have too many segments, let's say <=100k, there is no impacts. Otherwise , a small percentage of searches will complain: segments lacking.

Is there anyway to help mitigate this problem?

[yhmo]

In-memory replica can avoid the impact of such cases: https://milvus.io/docs/replica.md#In-Memory-Replica
The load_collection() has a replica_num parameter(default value is 1):
https://milvus.io/api-reference/pymilvus/v2.5.x/MilvusClient/Management/load_collection.md
Note:

1. to load N replicas, you must have at least N query nodes in a cluster.
2. to load N replicas, the total required memory is N times of one replica.

For example, you have 2 query nodes in a milvus cluster, and you load a collection with 2 replicas, each query node will load one replica. Once a query node is down, all the search/query requests will be processed by the other query node.

[xiaofan-luan]

Hey team, I have a question regarding bringing down and then bring up the query node. When I drain the EC2 node at EKS cluster, causing the query node brought down, the query node will release its segments through balancing process, and at the same time the rest of query nodes will load these segments.

Is this migration process able to guarantee that: during the whole process all searches towards these segments being migrated are not impacted?

Based on our observation, when the query node doesn't have too many segments, let's say <=100k, there is no impacts. Otherwise , a small percentage of searches will complain: segments lacking.

Is there anyway to help mitigate this problem?

I guess you need to tune the stop time .
by default it's several minutes so if the querynode can not drain its segment for more than this graceful timeout then the pod will be force killed.
This could happen when you have many segments and balance is slow.
Another option is to the balancer concurrency parameters, see if balance can be done faster

[xiaobingxia-at]

I have tuned the gracefullTimeout to 15 minutes, and can't be more than that. Because k8s will only wait until 15 minutes before killing the query node. That's why I was asking in this thread.

Regarding the balancer concurrency parameters, I only know these 3, anything else I should tune? @xiaofan-luan
checkBalanceInterval: 8000 # check balance every 8 seconds
collectionBalanceSegmentBatchSize: 30
collectionBalanceChannelBatchSize: 3