Memory-Aware Auto Load/Release Mechanism for Collections

[michaelsheka]

Hi Milvus team,

I'd like to propose a new feature that could enhance how Milvus handles memory and collection management.

Background
Currently, Milvus requires users to manually load and release collections, which can become cumbersome, especially in multi-tenant or memory-constrained deployments. If a collection is not loaded, queries or inserts will fail unless the user explicitly loads it.

Proposal
Introduce a memory-bound collection management mechanism where Milvus automatically handles loading and releasing collections based on system memory constraints.

The idea is:
When a user queries or inserts documents into an unloaded collection, Milvus checks:
If there is enough memory to load the collection.
If not, it automatically releases other collections (e.g., using an LRU as strategy to free memory).
Then loads the requested collection automatically.

Would love to hear your thoughts. I'm happy to elaborate further or help contribute if there's interest in this feature.

Thanks!

[yhmo]

So far as I know, the target of the next major version Milvus v2.6.x includes these points for loading:

• Segment lazy load: lazy loading allows for a mode where data can be stored in object storage, cached locally, and fetched from object storage when the local cache is missed.
• Tiered Storage: Lazy/Partial Load for raw data, scalar index and vector index.

The goal is to make the loading process more controllable and precise. In my opinion, if all these features are done, it is possible to achieve "automatically load/release collection".

Wait @xiaofan-luan to comment.

[michaelsheka]

Wow very quick reply. Thank you @yhmo.
I think that this is a big issue, think about classic RDBMS, you don't need to load or release tables - the RDBMS do it for you.

[yhmo]

Understand.

[xiaofan-luan]

Tiered storage might be something you need.
you can load everything but they are actually lazy loaded so don't take any resources.
When search happened, milvus will just load data from remote storage and cache it in memory or disk.
The search could be dramatically slow(like hundred milliseconds or even seconds) but it should be still faster than load/release and manage collection by yourself