Why is MinIO storage requirement so high for HNSW index type?

[abhinavkulkarni]

Hi,

According to the Milvus sizing tool, I would need 4x440 GB of MinIO storage for ingesting 100M vectors of 512 dimensions with an HNSW32 index built.

The MinIO storage requirement does not align with the size of raw and index data.

Each vector is 512*4 = 2KB in size. So 100M vectors would be 200GB in size.

According to this article, the index file size for an HNSW32 index should be 100M * 32 * 4 = 12.8 GB.

So, overall, the raw data + index file size is 212.8 GB.

With 4 pods of MinIO, erasure coding of 2 essentially means a replication factor of 2, so the total size would be 2x212.8 GB = 425.6 GB.

However, MinIO asks for 4x440 GB.

Why are MinIO requirements so high?

Thanks!

[yhmo]

So far as I know, we don't have an index type named "HNSW32". The available index types are listed here: https://milvus.io/docs/index.md?tab=floating#In-memory-Index
Do you mean the "HNSW_PQ"?

In cluster, minio/kafka/pulsar/etcd are supposed to be distributed services. minio service has its replica machinery. In our milvus-helm deployment, 4 pods are deployed for minio: https://github.com/zilliztech/milvus-helm/blob/1c053e888b855f72583b896b32eb5d2e457938f7/charts/minio/values.yaml#L101

Pulsar/kafka is like a WAL component for milvus. All the dml operations(insert/upsert/delete) are first stored into pulsar/kafka. So, once you have inserted 100GB data into milvus, there must be 100GB data in pulsar/kafka.

100M vectors of 512 dim, the size is 100M * 512 * 4bytes = 200GB
If you choose HNSW_PQ index type, the index size is about 1/4 of the original data size: 50GB

Milvus manages data by segments. When you continually insert data into milvus, there are small segments generated. Milvus internally triggers compact action to merge the small segments into large segments, here is an old article about this machinery: https://milvus.io/blog/2022-2-21-compact.md

After compaction, small segments will be marked as "soft-deleted", and wait for garbage collection to delete them. GC is triggered in an interval of a few hours. So the small segments still occupy disk space before they are deleted by GC.

Based on the points, the disk usage could be much higher than the original data size. It is recommended to assign 3X ~ 5X of disk space to avoid unexpected error of out of disk.

Our calculator's magnification ratio is a relatively conservative estimate, as it generally requires 3-5 times storage amplification under normal circumstances, including logs, raw data, and indexes. Nowadays, since disks are basically worthless, it is recommended to assign more disk space.

[xiaofan-luan]

Hi,

According to the Milvus sizing tool, I would need 4x440 GB of MinIO storage for ingesting 100M vectors of 512 dimensions with an HNSW32 index built.

The MinIO storage requirement does not align with the size of raw and index data.

Each vector is 512*4 = 2KB in size. So 100M vectors would be 200GB in size.

According to this article, the index file size for an HNSW32 index should be 100M * 32 * 4 = 12.8 GB.

So, overall, the raw data + index file size is 212.8 GB.

With 4 pods of MinIO, erasure coding of 2 essentially means a replication factor of 2, so the total size would be 2x212.8 GB = 425.6 GB.

However, MinIO asks for 4x440 GB.

Why are MinIO requirements so high?

Thanks!

the recommended sizing is fairly conservative.

Rememeber Minio need at least 2 replica and we would recommended to leave 20-30% buffer.

But yes I belive 4x300 GB disk space might works for your case as well.

[abhinavkulkarni]

So far as I know, we don't have an index type named "HNSW32"

@yhmo: Sorry, I meant HNSW with M=32.

If you choose HNSW_PQ index type, the index size is about 1/4 of the original data size: 50GB

Well, I'm choosing a simple HNSW index type with M=32, no product or scalar quantization.

According to the Faiss documentation, the index overhead per vector is M * 2 * 4 (basically two pointers per vector).

According to the Milvus documentation, the index overhead per vector is M * 4 (basically one pointer per vector).

the recommended sizing is fairly conservative.

@xiaofan-luan: Can you please explain again why you think this is conservative? Let me present the sizing calculations again:

Raw data size: 100M vectors of 512 dimensions = 200 GB
Index size (with M=32): 100M x 32 x 4 bytes per vector = 12.8 GB

So, total size = 212.8 GB.

Even with a MinIO replication factor of 2, the total size would be 2 x 212.8 = 425.6 GB.

Whereas the recommended size is 4x440 = 1760 GB.

Please let me know if I'm making any calculation mistakes.

Thanks!

[abhinavkulkarni]

I do understand that compaction and MinIO retention period before GC kicks in can add to size.

I also understand that in case of deletes, a segment can have 10% of the data marked as soft deletes, which will continue to occupy the space.

There are other things, such as Bloom Filters, scalar indexes, etc.

But the Milvus sizing of 1760 GB still seems like an overkill when the raw data + index file size adds up to 425.6 GB.

I'm sure I'm making calculation mistakes somewhere, so please let me know.

[abhinavkulkarni]

Please also note, if I build a FAISS index of type HNSW,Flat with M=32, the overall raw data + index size is around 10-20% more than the raw data size, which tracks with my calculations above.

Again, I understand Milvus is a distributed system, so size requirements are perhaps higher, why are they be so much higher?

[yhmo]

No no no.
In fact, the original vectors are stored into "segments", each segment might contain thousands of vectors.
Each segment has an independent index.
A HNSW index is stored in additional file, which contains all the vectors of a segment, and the edges between the vectors.

So, if a segment has one million vectors(128 dim), the original vectors are stored in one segment whose size is 1M * 128 * 4bytes = 512MB. The HNSW index file contains the 1M original vectors and M * 32 edges, the size of a HNSW index is 1M * 128 * 4bytes + 1M * 32 * 4bytes = 512MB + 128MB = 640MB

So,
Raw data size: 100M vectors of 512 dimensions = 200 GB
Index size (with M=32): 200 GB + 100M x 32 x 4 bytes per vector = 212.8 GB

So, total size = 412.8 GB.

[abhinavkulkarni]

Thanks @yhmo, that now makes sense.

Is there any particular reason why raw vectors are stored as part of index file (instead of index file storing only pointers into the raw data)?

Also, do both raw data file and index file get downloaded to querynode as well?