utility.wait_for_index_building_complete() return but index creation not finished?

[xu664]

Hello everyone, I want to test the time for creating index with 10 million data . I did something like this with pymilvus verison 2.5.6 and Milvus Distributed Mode

begin_time = time.time()
collection.create_index("vectors", index_params, index_name = "vector_index")
utility.wait_for_index_building_complete("xxxx","vector_index")
end_time = time.time()
print(f"total time create_index is {end_time - begin_time} s")

and it costs only 1.6 seconds! That is amazing. However if I immediately use utility.index_building_progress("xxx", "vector_index") , it shows

{'total_rows': 9999870, 'indexed_rows': 0, 'pending_index_rows': 9999870, 'state': 'Finished'}.

Therefore I am confused whether the index is actually created.

My question is

1. Why does utility.wait_for_index_building_complete("xxx", "vector_index") return when the creating index is not finished?
2. How to get the actual creating index time?
3. if I use collection.load() immediately after collection.create_index(), does it include creating index time?

[yhmo]

create_index() is called before insert():

import random

from pymilvus import (
    connections,
    FieldSchema, CollectionSchema, DataType,
    Collection,
    utility,
)

connections.connect(host='localhost', port=19530, user="root", password="Milvus")
print(utility.get_server_version())

collection_name = "CCC"
dim = 512

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=dim),
]
schema = CollectionSchema(fields)
utility.drop_collection(collection_name)
collection = Collection(collection_name, schema)
print(collection_name, "created")

collection.create_index(field_name="vector", index_params={
        "metric_type": "L2",
        "index_type": "IVF_FLAT",
        "params": {}
    })
print("index created")

for i in range(3):
    row_count = 10000
    data = [
        [k for k in range(row_count)], # id
        [[random.random() for _ in range(dim)] for _ in range(row_count)], # vector
    ]
    collection.insert(data=data)
collection.flush()
print("data inserted")


kk = utility.index_building_progress(collection_name=collection_name)
print("index progress", kk)

The index_building_progress() shows {'total_rows': 30000, 'indexed_rows': 0, 'pending_index_rows': 30000, 'state': 'Finished'}

index progress {'total_rows': 30000, 'indexed_rows': 0, 'pending_index_rows': 30000, 'state': 'Finished'}

create_index() is called after insert():

import random

from pymilvus import (
    connections,
    FieldSchema, CollectionSchema, DataType,
    Collection,
    utility,
)

connections.connect(host='localhost', port=19530, user="root", password="Milvus")
print(utility.get_server_version())

collection_name = "CCC"
dim = 512

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=dim),
]
schema = CollectionSchema(fields)
utility.drop_collection(collection_name)
collection = Collection(collection_name, schema)
print(collection_name, "created")


for i in range(3):
    row_count = 10000
    data = [
        [k for k in range(row_count)], # id
        [[random.random() for _ in range(dim)] for _ in range(row_count)], # vector
    ]
    collection.insert(data=data)
collection.flush()
print("data inserted")

collection.create_index(field_name="vector", index_params={
        "metric_type": "L2",
        "index_type": "IVF_FLAT",
        "params": {}
    })
print("index created")

kk = utility.index_building_progress(collection_name=collection_name)
print("index progress", kk)

The index_building_progress() shows {'total_rows': 30000, 'indexed_rows': 30000, 'pending_index_rows': 0, 'state': 'Finished'}

index progress {'total_rows': 30000, 'indexed_rows': 30000, 'pending_index_rows': 0, 'state': 'Finished'}

The state=Finished is not a reliable flag to know index work is done or not, it means the index is created and ready to build.

"indexed_rows" and "pending_index_rows" are the major values to observe index work.

[yhmo]

After all data is inserted, there might be compaction tasks pending, some small segments are indexed, but they need to be merged later. So, the index node will continue to build index for merged segments.
If indexed_rows is equal to the total number of entities, it means all the inserted data is indexed in some small segments. At this moment, the "pending_index_rows" is not zero, which means there are some larger segments generated(merged from small segments), index node will continue to build index for the large segments.
Only when pending_index_rows is zero, all the index tasks are done.