please help to see why this simple query returns no results

[bochen-usfoods]

I started a local instance (v2.5.10) with docker compose, followed https://milvus.io/docs/sparse_vector.md to insert and query sparse vectors. but this simple script returns no result.

from pymilvus import FieldSchema, CollectionSchema, DataType, MilvusClient

client = MilvusClient(uri="http://localhost:19530")

if client.has_collection("sparse_vector_example"):
    client.drop_collection("sparse_vector_example")
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="document", dtype=DataType.VARCHAR, max_length=1000),
    FieldSchema(name="sparse_vector", dtype=DataType.SPARSE_FLOAT_VECTOR)
]
schema = CollectionSchema(fields=fields)
index_params = client.prepare_index_params()
index_params.add_index(
    field_name="sparse_vector",
    index_name="sparse_inverted_index",
    index_type="SPARSE_INVERTED_INDEX",
    metric_type="IP",
    params={"inverted_index_algo": "DAAT_MAXSCORE"}
)
client.create_collection(
    collection_name="sparse_vector_example",
    schema=schema,
    index_params=index_params
)

sparse_vectors = [
    {1: 0.5, 100: 0.3, 500: 0.8},
    {10: 0.1, 200: 0.7, 1000: 0.9},
]
documents = ["Doc 1", "Doc 2"]
data = [
    {"document": doc, "sparse_vector": vec}
    for doc, vec in zip(documents, sparse_vectors)
]
client.insert(
    collection_name="sparse_vector_example",
    data=data,
)
print(f"Inserted {len(documents)} records into the collection")
client.load_collection(collection_name="sparse_vector_example")

search_params = {
    "params": {"drop_ratio_search": 0.2},
}
query_vector = [{1: 0.2, 50: 0.4, 1000: 0.7}]
print("Query vector:", query_vector)
res = client.search(
    collection_name="sparse_vector_example",
    data=query_vector,
    anns_field="sparse_vector",
    limit=3,
    output_fields=["document"],
    search_params=search_params,
)
print("\nSearch Results:")
print(res)
print("\nExample completed successfully!")

Output:

Inserted 2 records into the collection
Query vector: [{1: 0.2, 50: 0.4, 1000: 0.7}]

Search Results:
data: [[]]

Example completed successfully!

[yhmo]

The reason is: you call the search() immediately after insert(), the data is not fully consumed by querynode.
Milvus insert is a two-step operation:

1. The client calls insert() to pass the data to milvus proxy node, the proxy node receives the data and pass the data to a message queue(in standalone, the message queue is a RocksDB; in cluster, the message queue is pulsar/kafka) and then returns ok the client. At this moment, the data is in the message queue, waiting for querynode/datanode to consume, the insert() interface returns ok to the client. It is like a write-ahead log operation of the other databases.
2. After a few milliseconds, the data is consumed by the querynode/datanode, now the data is visible to search/query.

So, if you call search() immediately after insert(), the newly inserted data could be invisible to search/query, so that you get an empty result.
Milvus provides "consistency level" options for users:
https://milvus.io/docs/consistency.md#Consistency-Level

"Strong" guarantees the newly inserted data is visible to search/query, but it adds an extra 200ms latency to search/query.
"Bounded" guarantees that the inserted data earlier than a time window(by default, 5 seconds) is visible to search/query.
"Eventually" doesn't guarantee anything. search/query will be executed immediately regardless of whether the newly inserted data is visible or not.

So, you can try adding consistency_level="Strong" to the search request.

res = client.search(
    collection_name="sparse_vector_example",
    data=query_vector,
    anns_field="sparse_vector",
    limit=3,
    output_fields=["document"],
    search_params=search_params,
    consistency_level="Strong",
)

[bochen-usfoods]

yeah, it works, thanks for the explanation.