Loading collections / partitions dynamically on the server side.

[nairan-deshaw]

Hi,

For Milvus, we currently maintain the loading and releasing of partitions / collections on the client side and the applications have to ensure that memory limits are utilized efficiently by releasing unused collections. Is there a reason why this has not been done on the database side since the DB has better insights into the available memory and the collections that were accessed most recently.

We checked the roadmap items for Milvus and found Lazy Loading to be a similar feature where the server decides to load and release the collections based on the read operations from the user. Is that understanding right? If yes, is it going to be an LRU based eviction? Would be great if we can get a design proposal for this feature and the tentative timelines around when this feature is expected to be released.

Thanks.

[congqixia]

@nairan-deshaw yes, server do have a better insight which part of data is recently used and which part is not.
The previous Lazy load feature is at segment level which means and even at this level the cold segment could cause search/query performance drop dramatically if the search shall happens on a cold segment(which is lazy loaded and s3 or other oss operations is needed to make it accessible)
The team is working on a better design of the file format so that the memory usage and io could be reduce as much as possible

If the performance is not the major concern, lazy loading could be a viable choice to reduce memory usage but not tested thoroughly. We could discuss this feature in detail if you want to make a some concept test.

[nairan-deshaw]

Thanks @congqixia. We're performing searches wherein we have to load a partition, search it and then release it due to memory constraints. Also when there are multiple users, we have to ensure that loading and releasing partitions is done in a consistent manner. If this layer moves to the server, it simplifies a lot of client side interactions. We're looking for performance and memory is currently not a big concern and we can have higher values there. A good portion of the query time goes in checking the loading and release status of partitions, hence the issue.

Few questions:

• Basically, if we search a particular partition, does the Lazy loading feature ensure that it finds the right segment and loads it directly. Appreciate that this first load can have a high latency and every subsequent request should just use this pre loaded segment. How does it know which segment to load in a partition?
• What are the latency numbers we can expect if our underlying index files are stored on minio in the same K8s deployment?
• What eviction policies are used when a segment has to be unloaded?

[maggonravi]

@congqixia : do we have any timelines to implement this functionality on Milvus server end? It would really help us scale setup on our side.

[congqixia]

@maggonravi currently we are implementing the chunk-level cache for segments, which is the base for the feature. The plan is to have this ability after 2.6