Memory usage is high when indexes are loaded

[YolandaLyj]

I load 2kw lines of HNSW index, the memory usage during loading is twice that of loading completion, how can I reduce the memory during loading.

Below is my memory usage, which continues to rise as the index is loaded

[xiaofan-luan]

Loading data incurs some memory overhead due to additional copying and serialization/deserialization processes. Here are some suggestions to reduce memory consumption during loading:

Reduce segment size – This can lower memory usage but may also impact query performance after loading.
Adjust load concurrency – Tuning the concurrency level can help balance memory usage and performance.
Lower the pod memory limit – When Milvus reaches the memory limit, it dynamically reduces load concurrency to prevent excessive memory usage.
Upgrade to Milvus 2.5.6 – This version includes optimizations for memory and CPU efficiency during loading, with more improvements planned in future releases.

[YolandaLyj]

Thank you very much. I would like to ask you specifically how to adjust the loading concurrency. Only see the adjusted query (KnowhereInitSearchThreadPool) and build (KnowhereInitBuildThreadPool).

[yhmo]

The pool for loading index, its size is mainly affected by common.threadCoreCoefficient.middlePriority. Enlarge its value will increase the loading concurrency, but you need to be careful if you are using cloud storage since high concurrency might hit the quota limit.

milvus/internal/querynodev2/segments/pool.go

Line 115 in be86d31

poolSize := hardware.GetCPUNum() * pt.CommonCfg.MiddlePriorityThreadCoreCoefficient.GetAsInt()

milvus/configs/milvus.yaml

Line 811 in be86d31

middlePriority: 5 # This parameter specify how many times the number of threads is the number of cores in middle priority pool

[YolandaLyj]

Thank you very much!

[xiaofan-luan]

BTW, if you are using standalone, maybe the memory is consumed by index building and flush compaction, then we can not do too much about it