QueryNode Memory Overload Due to Unbalanced Growing Segments

[navneetrajput06]

Hi team,
We are encountering an issue where one of our QueryNodes becomes fully utilized in terms of memory (reaching 100%), while the others are underutilized. At one point, this caused our data ingestion to stop completely.
Using the following query:
sum(milvus_querynode_segment_num{segment_state="Growing"}) by (pod, segment_level)
we observed that a single QueryNode is holding 325 growing segments with segment_level="L1", whereas all other QueryNodes are showing zero growing segments.
Here are some relevant details about our setup:
• We have a collection with 4 shards and a total of 25 QueryNodes.
• Auto-balancing is enabled with the default configuration in QueryCoord.
• Restarting the affected QueryNode temporarily resolves the issue and ingestion resumes, but after some time, the imbalance returns.
We would appreciate any insights into why all growing segments are being assigned to just one QueryNode, and how we can ensure proper distribution to prevent ingestion failures.
Thanks in advance for your support.

[yhmo]

Which version of your milvus?

Use describe_collection() to double-check the shards_num value of the collection:

desc = milvus_client.describe_collection(collection_name)
print(f"{collection_name} :", desc)

How many data nodes in the cluster?
If shards_num is 4, there should be 4 query nodes to process growing segments, and 4 data nodes to persist segments.

You can install a monitoring service to observe the state of querynode/datacoord/datanode

[navneetrajput06]

@yhmo Milvus version is 2.5.4
Collection has 4 shards and total datanodes are 10 and we have more that 25 query node. As we have 4 shards growing segment atleast should distribute to 4 query nodes. Bug almost all growing segments going to one query node only.

[xiaofan-luan]

1. please upgrade to 2.5.13 and it should fix several critical issue.
2. If you have 4 shards, then growing data should be assigned to 4 querynodes yes you are right. I believe upgrade to 2.5.13 should fix that issue.
3. after you upgrade , you could still see the delegator memory is much higher than the other node. then you can tune querycoord.delegatorMemoryOverloadFactor parameter to higher (By default is 0.1, upper limit is 1.0, recommended to change to 0.5 and see if it's working).

[navneetrajput06]

@xiaofan-luan one observation i would like to add is the problematic node have growing segments live for hours where else in other hosts growing segments are finishing and new are being assigned to them. This problematic node keep assigned with new segments but not finishing previous.

[xiaofan-luan]

try 2.5.13 see it solved the issue. I know couple of compaction stuck issue , which might be related to the issue you hit