milvus_datacoord_stored_rows_num and milvus_querynode_entity_num metrics are not synchronizing

[navneetrajput06]

milvus_datacoord_stored_rows_num and milvus_querynode_entity_num metrics are not synchronizing at any point of time even after stopping upset operations on collection.

As per documentation, When you perform an upsert operation, it will overwrite (delete and replace) an existing entity if a specified field already exists in the collection.
• When data is deleted, entities are only marked as "logically deleted"
• Space is freed through two processes:
◦ Compaction: Automatically runs in the background to merge smaller segments and remove logically deleted data
◦ Garbage Collection: A separate process that periodically removes "Dropped" segments to free up storage space

So deleted rows from upsert operations will eventually be cleaned up through these background processes, but there may be a delay between deletion and actual space reclamation.

However, even after stopping all operations on the collection for over 20 days, these two metrics have not aligned and continue to show different values — although they ideally should converge over time.

Could you please highlight the reason for this scenario?

[yhmo]

Which version of your milvus?
The milvus v2.5.3 fixed a bug that deleted data might be lost due to clustering compaction

• Fixed a potential loss of deleted data due to clustering compaction. (#39133)

The milvus v2.5.8 fixed another bug about delete:

• Fixed potential data loss in delete operations caused by duplicate binlog IDs (#40985), (#40976)

[navneetrajput06]

When i did flush it all came equal. could you please explain this. we need a metrics to show quaryable and stored rows comparison.

[yhmo]

Use this script to observe:

import random
import time
import numpy as np
import requests

from pymilvus import (
    MilvusClient, DataType,
)

def gen_embedding():
    return [random.random() for _ in range(dim)]


def print_metrics():
    print("======================================================================")
    resp = requests.post(url="http://localhost:9091/metrics", json={}, timeout=5)
    for line in resp.iter_lines(chunk_size=10):
        aa = line.decode('utf-8')
        if aa.startswith("#"):
            continue
        if (("milvus_datacoord_stored_rows_num" in aa)
                or ("milvus_querynode_entity_num" in aa)
                or ("milvus_proxy_upsert_vectors_count" in aa)):
            print(aa)
    print("======================================================================")


client = MilvusClient(uri="http://localhost:19530")
print(client.get_server_version())

collection_name = "AAA"
dim = 384

schema = MilvusClient.create_schema(enable_dynamic_field=False)
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=dim)

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="vector",
    index_type="HNSW",
    metric_type="COSINE",
    params={"M": 8, "efConstruction": 100},
)

client.drop_collection(collection_name=collection_name)
client.create_collection(
    collection_name=collection_name,
    schema=schema,
    index_params=index_params,
)
print(collection_name, "created")

for i in range(10):
    data = [{"id": k, "vector": gen_embedding()} for k in range(10)]
    client.upsert(collection_name=collection_name, data=data)
    print(collection_name, "upserted")
    print_metrics()
    time.sleep(1)

print("all upsert done")
for i in range(10):
    print_metrics()
    time.sleep(1)

client.flush(collection_name=collection_name)
print(collection_name, "flushed")
for i in range(10):
    print_metrics()
    time.sleep(1)

client.compact(collection_name=collection_name)
print(collection_name, "compacted")
for i in range(10):
    print_metrics()
    time.sleep(1)

Eventually, the milvus_datacoord_stored_rows_num/milvus_proxy_upsert_vectors_count/milvus_querynode_entity_num come to the same value.

Flush operation forces the datanode to persist data, so the milvus_datacoord_stored_rows_num becomes 100 after flush().

[navneetrajput06]

@yhmo thanks for quick response. here in this case we are calling flush and Compaction manually thats the reason counts are matching.
In live Scenario, L0 segments are creating and they doesn't flushes automatically for days as a result these metrics are not coming same at any point of time.
Is there any configuration which blocks this to flush?

[yhmo]

It might be a bug if L0 compaction is stuck. Similar issue: #41020
Recent versions of v2.5.x fixed a couple of compression bugs. If your Milvus version is old, you can try upgrading to v2.5.8.

[yhmo]

I just talked with @XuanYang-cn, it is a normal case that the milvus_datacoord_stored_rows_num can be different from milvus_querynode_entity_num.

milvus_querynode_entity_num is the actual number of entities.
milvus_datacoord_stored_rows_num depends on the compaction policy. Not each delete action can trigger compaction. Assume a collection has a segment containing 100 rows, now you call delete() to delete one entity. The milvus_querynode_entity_num is 99. But the milvus_datacoord_stored_rows_num is 100. The reason is: it is unnecessary to compact the segment since only one entity is deleted, the policy requires the deleted entities to exceed a threshold (10% or 20%). Now even if you stop all operations for over 20 days, the milvus_datacoord_stored_rows_num is always 100.

So, we don't promise that the milvus_datacoord_stored_rows_num must be equal to milvus_querynode_entity_num.

[navneetrajput06]

@yhmo @XuanYang-cn in that if i call compaction manually it should be compacted with uniq rows. its not even happening when i am calling compaction manually. When i say compact L0 segments stay there and they are not compacting to L1. but when i called flush it went fine both metrics came as same number of rows.