Get RPC Error 'Timeout was reached' when loading collection

[CodinSheep]

Hi Team!
I am new to milvus. I've deployed a 2.5.x cluster. Inserting data and building index are fine. A 100M dataset, dimention=96 and HNSW index.
But when I try to load the collection, a timeout error often occurs, like:

Failed to load collection: deep100M_HNSW
RPC error: [get_loading_progress], 
<MilvusException: (code=2018, message=show collection failed: failed to load field data: Error in GetObjectBuffer[errcode:-1, exception:, errmessage:curlCode: 28, Timeout was reached, params:params, bucket=my-release, object=files/insert_log/457019981266969131/457019981266969132/457019981401121734/0/457019981401169293])>, 
<Time:{'RPC start': '2025-03-31 14:58:20.645231', 'RPC error': '2025-03-31 14:58:22.876356'}>
RPC error: [wait_for_loading_collection], 
<MilvusException: (code=2018, message=show collection failed: failed to load field data: Error in GetObjectBuffer[errcode:-1, exception:, errmessage:curlCode: 28, Timeout was reached, params:params, bucket=my-release, object=files/insert_log/457019981266969131/457019981266969132/457019981401121734/0/457019981401169293])>, 
<Time:{'RPC start': '2025-03-31 14:24:12.826746', 'RPC error': '2025-03-31 14:58:22.877739'}>

I've oberved that the memory usage of the nodes increased (about 30GB), stayed still for a while and this error happened. I retried it for several times (release and load again), the collection finally become loaded. And the search operation is good.
So I guess the load process is working well ? Is there any component that is responding too slowly causing this error ? Are there any adjustable parameters that may help ?
Here is my cluster (The index is done so I limit the number of index node to 1)

Any help or explanation will be very appreciated, THANKS !

[yhmo]

"Timeout was reached" ----------- This error was thrown from S3Client.GetObject api.
Possible reason: the minio service is in high traffic.

Try increasing the timeout value minio.requestTimeoutMs:

milvus/configs/milvus.yaml

Line 156 in 216be14

requestTimeoutMs: 10000 # minio timeout for request time in milliseconds

Or decrease the common.threadCoreCoefficient.middlePriority to lower the concurrence of loading chunk files:

milvus/configs/milvus.yaml

Line 811 in 216be14

middlePriority: 5 # This parameter specify how many times the number of threads is the number of cores in middle priority pool

[CodinSheep]

Thanks! I'll have a try.

[xiaofan-luan]

Hi Team! I am new to milvus. I've deployed a 2.5.x cluster. Inserting data and building index are fine. A 100M dataset, dimention=96 and HNSW index. But when I try to load the collection, a timeout error often occurs, like:

Failed to load collection: deep100M_HNSW
RPC error: [get_loading_progress], 
<MilvusException: (code=2018, message=show collection failed: failed to load field data: Error in GetObjectBuffer[errcode:-1, exception:, errmessage:curlCode: 28, Timeout was reached, params:params, bucket=my-release, object=files/insert_log/457019981266969131/457019981266969132/457019981401121734/0/457019981401169293])>, 
<Time:{'RPC start': '2025-03-31 14:58:20.645231', 'RPC error': '2025-03-31 14:58:22.876356'}>
RPC error: [wait_for_loading_collection], 
<MilvusException: (code=2018, message=show collection failed: failed to load field data: Error in GetObjectBuffer[errcode:-1, exception:, errmessage:curlCode: 28, Timeout was reached, params:params, bucket=my-release, object=files/insert_log/457019981266969131/457019981266969132/457019981401121734/0/457019981401169293])>, 
<Time:{'RPC start': '2025-03-31 14:24:12.826746', 'RPC error': '2025-03-31 14:58:22.877739'}>

I've oberved that the memory usage of the nodes increased (about 30GB), stayed still for a while and this error happened. I retried it for several times (release and load again), the collection finally become loaded. And the search operation is good. So I guess the load process is working well ? Is there any component that is responding too slowly causing this error ? Are there any adjustable parameters that may help ? Here is my cluster (The index is done so I limit the number of index node to 1)

Any help or explanation will be very appreciated, THANKS !

Seems that you are trying to run 100M data on a single machine minio?

I think the problem is your object storage can not offer enough iops to load.

I would recommended not to use single machine minio on a distributed milvus. And by default is should be always 4 node. you can scale up minio to get better QPS.

Also if you running on cloud, the best you can do is running it on S3/GCS or OSS since cloud object offers unlimited throughput

[CodinSheep]

Thank you very much! But I'm not quite clear on how to scale the MinIO service to multiple machines.

The cluster is 1master + 3worker. The current MinIO runs only on the master node using a default k8s Storage Class based on NFS.

If I create more MinIO replica, do they need local storage of the machine they are running on, or, using the storage class on NFS is enough?

[xiaofan-luan]

NFS is usually slow, that's why you hit timeout issue.

I would suggest to use minio + local disk or skip WAL and use NFS directly by mounting nfs as persistent volume and let milvus run on local storage mode

[xiaofan-luan]

The other way to fix that is to reduce the load threadpool, but this is not recommneded