I want to know about the mmap structure.

[ian-lss]

Due to memory capacity issues, I upgraded from Milvus 2.3.12 to 2.4.4.
I changed the vector index type from SCANN to HNSW, and after the upgrade, the memory usage significantly decreased from 10G to about 1.7G, which is helpful.

However, I find something strange.
I understand that mmap stores pointer information of the file in memory, and the actual data is stored on disk instead of in memory. But currently, the total file size of mmap is about 200k.

Does this mean mmap does not store data in the file or that it stores data node and index node information? Is it possible that the physical memory allocation is still the same 13G as before the upgrade, and because there is plenty of free memory, it does not store data in the mmap directory?

Or should I understand it as something similar to the buffer pool (db cache) in an RDBMS? In other words, does it load data up to the capacity of physical memory, and when the load exceeds the physical memory capacity, it writes some of the existing memory data to the mmap directory?

[yhmo]

If mmap is enabled, when you call collection.load(), the query node will download the index files to local. The default local path is configured by the localStorage.path + "/mmap" in the milvus.yaml. And maps the index files into memory, but the files are not immediately read from disk to physical RAM, so the memory usage is very low. The index files are read in a lazy manner.
I don't think there is extra information in the mmap, it simply maps the index files between disk and memory.

[zhu3359]

If mmap is enabled, when you call collection.load(), the query node will download the index files to local. The default local path is configured by the localStorage.path + "/mmap" in the milvus.yaml. And maps the index files into memory, but the files are not immediately read from disk to physical RAM, so the memory usage is very low. The index files are read in a lazy manner. I don't think there is extra information in the mmap, it simply maps the index files between disk and memory.

@yhmo when i call collection.release(), will query node delete the disk files? or just erase the collection in memeory? dose mmap configure make some difference?

[yhmo]

The mmap files will be erased only after the size of mmap files exceeds this threshold:

queryNode:
  mmap:
    maxDiskUsagePercentageForMmapAlloc: 50 # disk percentage used in mmap chunk manager

[ian-lss]

Thank you for your response.
The collection is already loaded.
I think the index file you mentioned refers to the index file of the index node.
I don’t understand the meaning of

in a lazy manner.

Does it mean that the index file of the index node is downloaded over time?
Or does it mean that the index file is downloaded to the mmap directory when a search is performed?
The size of the mmap directory has not increased for almost 10 hours after calling collection.load(), and searches are currently being conducted in the service.
Only the memory usage is increasing, but again, the size of the mmap directory is not increasing.
The memory usage has not yet exceeded the physical memory size.
I did not activate mmap per collection, but I activated it in the config (.yaml).

[yhmo]

Index node builds index, uploads index files to S3.
When the collection is loaded, query node downloads the index files to local path.
Query node maps the file between RAM and disk by mmap().
Query node calls unlink() to drop the index file. But the mmap file still occupies disk space.
When search, query node actually read data from disk, acts like read from RAM.

[ian-lss]

@yhmo
Thank you for your response.
I understand that the index files are downloaded and mapped through mmap() during the load process, and then the index files are deleted.
However, you mentioned that the mmap files still occupy disk space, but in the mmap directory, as shown in the image below, there are only directories and no files.
Where are the mmap files stored?
Also, you mentioned that data is actually read from the disk. Does this mean it reads the mmap files, or does it read data stored somewhere in the query node?
The mmap files don’t seem to be present in the image, and I couldn’t verify the stored data.
The reason for my question is that I expect files corresponding to the memory usage before mmap was used to be stored on the query node disk, but I can’t see such files.

[yhmo]

The directories are empty but the data is still on disk. The file is invisible since unlink().
If you want to see the files, wait a few versions. The engineer will make some changes to keep the files visible.

[ian-lss]

@yhmo
Thank you very much for your response.
I understand about unlink() now.
I look forward to the additional features.

Thank you again for the explanation.

[navneetrajput06]

@yhmo could you please help us to get the below answers.

Current Setup Overview
• 865 + collections in Milvus and 260+ in a db.
• In a DB with 260+ collections— many collection has over 1 million records, with some having 10+ million . Total records in DB is more that 50M.
• 24 query nodes with resource: 32 CUP, 384 memory.
• All collections are loaded into memory
• Each collection has one shard and one partition
• Query nodes are experiencing ephemeral storage issues (had to increased volume attached to 200GB earlier it was failing because of ephemeral storage threshold exeeded.
• Mmap is enabled.

What could be the potential issue here and what can be tuned. also please help us to get the answer for below questions.

1. tried to check space utilization in our query node but do not have access to go into the sub directories.
2. What data is being written on the local disk and who is writing that.
3. what type of data it is.

[yhmo]

As my mention in this comment: #33621 (reply in thread)

"Query node calls unlink() to drop the index file. But the mmap file still occupies disk space."

With unlink(), the files are invisible to "du" or "df", but they still occupy disk space.

[yhmo]

This tool can help estimation: https://milvus.io/tools/sizing
If you are using mmap, check the "Offloading Fields to Disk" box, then you can see each query node requires local disk.

[navneetrajput06]

@yhmo thanks for sharing this i got your point completely. As per this tool the local disk utilization should be zero if we are not using mmap. We enabled mmap just to check. originally ephemeral storage threshold exceeded without enabling mmap. our querynode was consuming more than 40GB ephemeral storage.

[yhmo]

So far as I know, only mmap and local logs can occupy disk for query node. If mmap is disabled, the scalar fields data is cached into RAM memory.

[xiaofan-luan]

@yhmo thanks for sharing this i got your point completely. As per this tool the local disk utilization should be zero if we are not using mmap. We enabled mmap just to check. originally ephemeral storage threshold exceeded without enabling mmap. our querynode was consuming more than 40GB ephemeral storage.

scalar datas may also mmaped to local disk if you config like that

[czs007]

see alse #41636