datacoord remote_chunk_manager walk through object failed

[rere950303]

• milvus version: v2.4.20
• mode: cluster

---

Occasionally, we see a warning log like the one below. Is it okay to ignore it? Could this warning indicate a potential issue with Milvus operations or s3 storage?

[storage/remote_chunk_manager.go:320] ["failed to walk through objects"] [prefix=milvus/insert_log] [recursive=true] [error="Get \"http://s3host/milvus/?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix=milvus%2Finsert_log\": net/http: timeout awaiting response headers"]

[yhmo]

I just browsed the source code, this warn log is printed in the milvus garbage collection loop. GC action is triggered with an interval of 1 hour. GC action recursively checks all objects under the insert_log path to see if there are segments that can be cleaned up.

Seems it time out in the s3 client ListObjects() interface. By default, s3 client has a default setup with default responseHeaderTimeout = 1 minute and maxKeys = 1000. With maxKeys=1000, ListObjects() will return 1000 items for each call, maybe sometimes it timeout due to network fluctuation.

You can decrease this value to a small value(50 or 100):

milvus/configs/milvus.yaml

Line 159 in b03e605

listObjectsMaxKeys: 0

listObjectsMaxKeys=0 means using the default maxKeys for ListObjects().

[rere950303]

@yhmo
Thank you for the detailed explanation.

• If we're not using MinIO as the S3 storage, will setting the minio.listObjectsMaxKeys key in the values.yaml file still be applied? Or does it need to be applied in the config.tpl instead?

• And after changing that setting and running a Helm upgrade, do we need to restart the datacoord pod?

[yhmo]

remote_chunk_manager is a unified interface for all blob storage clients. The "minio" configurations are for all the supported clients. You can read the description of cloudProvider:

milvus/configs/milvus.yaml

Line 146 in b03e605

cloudProvider: aws

[rere950303]

@yhmo
Got it. I’ve confirmed that the related pods restarted after updating minio.listObjectsMaxKeys to 100 in config.tpl in helm. Please let me know if there are any issues with this change.