Can mmap be used when building a HNSW index?

[drjtan]

I understand that with mmap support, loading an existing HNSW index for search reduces memory usage a lot. But can milvus use mmap during insertion? If not, is DiskANN the only option if the server doesn't have enough RAM to build the index?

[yhmo]

mmap only takes effect for index files and raw data files.
Insert actions will generate sealed-segments, raw data files are stored in s3//minio in the sub-dir of each sealed-segments. Index node builds index for sealed-segments. Once an index is finished, the index files will be stored in s3/minio.
If a collection's "mmap.enabled" is set to true, for the sealed-segments that has no index, the query nodes will load its data files with mmap; for the sealed-segments that has index, the query nodes will load its index files with mmap.

[xiaofan-luan]

I understand that with mmap support, loading an existing HNSW index for search reduces memory usage a lot. But can milvus use mmap during insertion? If not, is DiskANN the only option if the server doesn't have enough RAM to build the index?

Milvus split data into segments, by default it's 1G. Which means even if you have 10G data, buidling index will only takes you 2-3 GB memory instead of 10GB+

[xiaofan-luan]

if you don't have enough memory, the easiest way it to reduce the segment size to 256MB(of course it will affect your performance)