Out of memory in proxy pod container

[jubingc]

Hi,

Our proxy pod containers were killed and restarted due to being out of memory. We were seeing logs from the proxy container.

Task fd452a0ba223e52074665cc3cbef2db605ea78b37612c148dbb9b8e12f59d7ea ran out of memory

I tried to limit the memory usage of the proxy pod so that the proxy pods don't get killed but I couldn't find any related configuration on the page https://milvus.io/docs/configure_quotaandlimits.md.

Maybe it is just my overlook. Can I have some pointers?

[xiaofan-luan]

how large is the memory?
proxy usually don't spend too much memories, only if:

1. very large nq or topks
2. too much request in the queue.

You should see some indicators in your monitorning metrics.

To fix that,
I think we need to limit the size of request queue in proxy, not only by request number (right now there is a config to tune it to smaller), but also limit the request size and response size.

@weiliu1031 can you help on it?

[jubingc]

Yes, you are right. It is not normal usage. Users tried to pull over 1m entities in a query and had hundreds of queries like that. Each query consumes about 200MB of memory in a proxy pod. Tens of concurrent queries are enough to kill the proxy pod container. We have encouraged users to use aggregation and pagination instead and have not seen the same issue so far. But we need a way to protect milvus itself from unexpected use.

[weiliu1031]

Milvus provides a way to limit the result size of each query through the parameter quotaAndLimits.limits.maxOutputSize, which defaults to 104,857,600 bytes.

Additionally, Milvus allows you to control the maximum number of concurrent queries on the proxy using the parameter proxy.maxTaskNum, with a default value of 1024.

In your case, adjusting both the maximum result size per query and the concurrency limit can effectively help manage the proxy's memory consumption during query execution.

In the future, if there is a clearer demand, we may consider setting more adaptive default values for these parameters based on the proxy’s memory size.

Hope this helps! If you encounter any issues, feel free to reach out to us

[jubingc]

Hi @weiliu1031

Thanks for your pointer. It seems that both quotaAndLimits.limits.maxOutputSize and proxy.maxTaskNum are not documented in https://milvus.io/docs/configure_quotaandlimits.md.

I only can find them from the source

milvus/pkg/util/paramtable/quota_param.go

Line 1640 in 16efcda

Key: "quotaAndLimits.limits.maxOutputSize",

and

milvus/pkg/util/paramtable/component_param.go

Line 1597 in 16efcda

Key: "proxy.maxTaskNum",

.

I will try them out. Thanks!

[weiliu1031]

Thanks for your feedback! We’ll update the documentation accordingly.

[yhmo]

Which version of your milvus? I didn't see any source code throw log like "Task xxxxx ran out of memory".

You can decrease the proxy.maxTaskNum value in milvus.yaml to reduce the OOM risk of proxy:

proxy:
  maxTaskNum: 1024

And, you can deploy more proxy nodes to balance the requests.

[jubingc]

The milvus version is v2.5.5. The log was from proxy container events and should be emitted by k8s.