Q&A Why does DiskANN Index consume much memory?

[yongbin813]

In 2.2.9 Standalone version , I have a collection ,use DiskANN index build 10 Million rows 512 Dimensions , according to https://milvus.io/tools/sizing will use 4.8 G Memory , but in reality, it was used 16G+ Memory , I was confused

This is my milvus.yaml:

queryNode:
cacheSize: 2 # GB, default 32 GB, cacheSize is the memory used for caching data for faster query. The cacheSize must be less than system memory size.
port: 21123
loadMemoryUsageFactor: 3 # The multiply factor of calculating the memory usage while loading segments
enableDisk: true

segment:
maxSize: 2048 # Maximum size of a segment in MB
diskSegmentMaxSize: 2048

DiskIndex:
MaxDegree: 28
SearchListSize: 16
PQCodeBudgetGBRatio: 0.058
BuildNumThreadsRatio: 1.0
SearchCacheBudgetGBRatio: 0.01
LoadNumThreadRatio: 8.0
BeamWidthRatio: 4.0

[yhmo]

Is there any long-length VARCHAR type field in the collection?
And, did you set the "replica_number" value when you call load()? Like this collection.load(replica_number=2)

[yongbin813]

As you said wait_for_index_building_complete(collection_name) After the end, the memory will indeed decrease, but there is still a difference with https://milvus.io/tools/sizing , 10 Million can use 12G memory! Is there any other configuration that I ignored?

[yhmo]

I think it is caused by the milvus compaction actions and index tasks. Small segments are compacted to large segments, and index node build index for large segments. If the large segment size is still smaller than the segment.maxSize, it will continue to compact them.
Since your milvus is a standalone, all the components(index node, data node, query node) are running in one process. Compaction is executed by data node, index is built by index node, data is loaded by query node, all of them consume memory. Memory usage will decrease until all compaction actions and index tasks are completed.

wait_for_index_building_complete() ensures that a "data snapshot" is fully indexed.

[luhuaei]

Hi, @yhmo Is there any way to limit the memory used for building an index? It's okay if it takes longer, as my available memory is limited.

[yhmo]

Memory usage of building index mainly depends on the size of a segment. Max size of segment is configured in the milvus.yaml:

dataCoord:
  segment:
    maxSize: 1024 # The maximum size of a segment, unit: MB. datacoord.segment.maxSize and datacoord.segment.sealProportion together determine if a segment can be sealed.
    diskSegmentMaxSize: 2048 # Maximun size of a segment in MB for collection which has Disk index

For in-memory indexes, segment size is configured by dataCoord.segment.maxSize, default value is 1024MB.
For diskann index, segment size is configured by dataCoord.segment.diskSegmentMaxSize, default value is 2048MB.
You can change them to smaller value according to your index type.

Note: with small segment size, the search performance will be slower than large segment.

[luhuaei]

Thank you very much for the explanation.