Query node and caching.

[Sarthak2000]

As per my understanding:

1. Query Node Chunk cache is only used to download the original vector data into the local disk from remote to speed performance.
   Q: What if my local disk size doesn't contain enough storage? Eg: If my remote size is 100GB but my local disk is 10GB will it still serve to queries by fetching 10GB from local disk and search the remaining from remote or would just fail?

2. When vectors are inserted, the inserted data will be accumulated into querynode's memory.
   Q: So if I am inserting 100GB data with 32GB query node's RAM will I have to keep doing a memory.release in order to avoid any error message of the memory getting full (error reports like "memory usage in high level, all dml requests would be rejected")?.

3. Milvus v2.x doesn't support swapping data between memory and disk. It requires the target collection must be fully loaded into memory before searching.
   Q: Does it mean all segments of that collections need to loaded to memory? So my memory size must be same as disk size?

4. Q: Does Milvus support caching so that repeated queries are not sent to knowhere engine but are handled at query node itself. By that I mean if multiple queries which are in similar nature, could they be handled in query node itself instead of it repeatedly going to knowhere to perform a disk based search?

[yhmo]

1. In old versions, in the milvus.yaml, the queryNode.cache.memoryLimit is used to limit the disk usage of chunk cache. If the remote size is larger than the limit, old files will be removed for downloading new files.
   In new versions(v2.5.x), the queryNode.cache.memoryLimit is marked as deprecated, not used anymore. QueryNodes download files only when mmap is enabled.

   milvus/configs/milvus.yaml

   Line 439 in 18625d7

   memoryLimit: 2147483648 # Deprecated: 2 GB, 2 * 1024 *1024 *1024

   
   If mmap is disabled, querynodes will directly read the remote files into memory.
   If mmap is enabled, the max disk usage of mmap is defined by queryNode.maxDiskUsagePercentage, if the disk usage exceeds this value, you will get an error: "Failed to create a new mmap_block, not enough disk for creating a new mmap block."

2. Only after you call load_collection(), a collection is in "loaded" state, the newly insert data is accumulated into querynode's memory. So, if you want to insert 100GB of data but only have 32GB RAM of querynodes, milvus will deny any insert requests when the inserted data exceeds 32GB. This logic is equal to: you have a collection with 100GB of data(index size=100GB), you want to load it in 32GB RAM of querynodes, milvus will fail to load it and return an error "load segment failed, OOM if load".

3. Milvus manages data by segments, each segment has an independent index. By default, a segment size is around 1GB, so there might be 100 segments for 100GB of data. Querynodes always try to load index of each segment. If a segment's index is not ready, querynode will load its original data. Different index types have different index sizes. For example, size of IVF_FLAT/HNSW index is a bit larger than the original data, size of IVF_SQ8/IVF_PQ is about 25% ~ 30% of the original data, size of DISKANN is about 1/6 ~ 1/4 of the original data. So, if all the segments are fully indexed, the required memory size of querynodes mainly depends on the index types.

4. Milvus doesn't cache any results for query/search, you can do this in the upper layer applications.

[xiaofan-luan]

As per my understanding:

1. Query Node Chunk cache is only used to download the original vector data into the local disk from remote to speed performance.
   Q: What if my local disk size doesn't contain enough storage? Eg: If my remote size is 100GB but my local disk is 10GB will it still serve to queries by fetching 10GB from local disk and search the remaining from remote or would just fail?
2. When vectors are inserted, the inserted data will be accumulated into querynode's memory.
   Q: So if I am inserting 100GB data with 32GB query node's RAM will I have to keep doing a memory.release in order to avoid any error message of the memory getting full (error reports like "memory usage in high level, all dml requests would be rejected")?.
3. Milvus v2.x doesn't support swapping data between memory and disk. It requires the target collection must be fully loaded into memory before searching.
   Q: Does it mean all segments of that collections need to loaded to memory? So my memory size must be same as disk size?
4. Q: Does Milvus support caching so that repeated queries are not sent to knowhere engine but are handled at query node itself. By that I mean if multiple queries which are in similar nature, could they be handled in query node itself instead of it repeatedly going to knowhere to perform a disk based search?

1. this is not gonna to work. Milvus now has to cache all the data. Milvus 2.6 will have a cache mode and will be able to evict data from local disk to remote storgae(Might be very slow)

2. we do swap between memory and local disk. search for mmap in the documents.

3. most of the query vectors are different(even though they might me similar) Is there a specific reason caching search result might be helpful?