about load collection

[huangxueliang]

when we use the vector, need to load data(load collection -c cc_test) into memory,if there are large data beyond the Mem capacity,how can we do?

[yhmo]

load() will return an error saying "service resource insufficient".
The required memory size mainly depends on data size and index type.

You can estimate the original data size in this way: each dimension of vector is represented as a float32 value, which occupies 4 bytes. So, the original data size is: row_count * dimension * 4 bytes
Typically, the index size compare to the original vectors size:

• IVF_FLAT/HNSW, a bit larger than the original vectors size
• IVF_SQ8/IVF_PQ, 25% ~ 30% of the original vectors size.
• DISKANN, 1/4 ~ 1/6 of the original vectors size.

If you estimate the data size is 30GB, you try to use a 32GB machine, it could be risky since sometimes memory usage peak may exceed the max capacity and cause OOM. It is recommended to keep more free space for memory.