Get Index Size of individual vector fields

[flippy1313]

In Milvus 2.5.12, I have collections with multiple vector fields. I use
client.load_collection(
collection_name=collection_name,
load_fields=["id",one_vector_field_name]
skip_load_dynamic_field=True
)
And I want to see the memory consumption of loading the column with a specific index. However, the memory consumption of the milvus process even without other jobs is very unstable. For example, the same ivfpq index has occupied 10 - 40GB on the same field on different loads. Is there a way to directly get the size of the index on that specific column?
In addition, this might be a bug on load_fields. Sometimes, when I try to load a large hnsw index, the load status returned as but I cannot query the collection. The error says that field is not loaded.

[yhmo]

Index size in theory:

nb ------ numer of rows
d -------- vector dimension
m, nbit, nlist ------ index parameters

"the same ivfpq index has occupied 10 - 40GB on the same field on different loads" ---------------- Possible reason: the vector field is not fully indexed. Milvus manages data by segments, with each segment having its own index. Index nodes build these indexes for segments asynchronously, one by one. After you call load(), the query node loads segments sequentially. If a segment has not been indexed yet, the query node will load the original vector data to perform a brute-force search. The theoretical size of IVF_PQ is about 25% to 30% of the original vector size. So, if the original vector size is 40GB, you will see 10GB of memory occupied if all segments are indexed. However, if none of the segments are indexed, 40GB of memory will be occupied.

"Sometimes, when I try to load a large hnsw index, the load status returned as but I cannot query the collection. The error says that field is not loaded." -------------------- Could you offer more info so that we can reproduce it?

[xiaofan-luan]

In Milvus 2.5.12, I have collections with multiple vector fields. I use client.load_collection( collection_name=collection_name, load_fields=["id",one_vector_field_name] skip_load_dynamic_field=True ) And I want to see the memory consumption of loading the column with a specific index. However, the memory consumption of the milvus process even without other jobs is very unstable. For example, the same ivfpq index has occupied 10 - 40GB on the same field on different loads. Is there a way to directly get the size of the index on that specific column? In addition, this might be a bug on load_fields. Sometimes, when I try to load a large hnsw index, the load status returned as but I cannot query the collection. The error says that field is not loaded.

I would be really helpful if you offer following information

1. your schema
2. your index params
3. the size of your data, and the load memory usage
4. logs of your cluster
5. if mmap is enabled