Why does performance degradation and collection loading & release failure occur due to entity upsert? #40408
Answered
by
yhmo
pipi-olo
asked this question in
Q&A and General discussion
-
|
Hi. When using entity upsert, Empty segments (L0) are created in the collection. As a resut, Timeout occur during Milvus collection loading and release. DataCoord Logs QueryCoord Logs Birdwatcher cf. I have removed the logs for the other collections. Q1. Why does empty segments (L0) appear when performing an entity upsert? |
Beta Was this translation helpful? Give feedback.
Answered by
yhmo
Mar 6, 2025
Replies: 1 comment 5 replies
-
|
upsert = delete + insert Datanode triggers compaction actions to erase entities by the IDs from L0 segment. The empty L0 segments mean their IDs have been processed. Empty L0 segments will be remove by GC(garbage collection) of milvus. GC action is triggered with an interval of 1 hour. Intensive/continually upsert action will downgrade search performance because it is a heavier operation than normal insert. The log doesn't show the root cause why load() timeout. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Thanks. Does upsert not check whether the data exists? Does the delete operation always occur? Do empty L0 segments occur due to the mandatory delete operations? |
Beta Was this translation helpful? Give feedback.
-
|
upsert doesn't check data existence, because with more and more data inserted, the time cost to check existence becomes longer and longer, and eventually unbearable. The birdwatcher doesn't count the real row count of L0 segment, so you see the "Row Count" of L0 is 0. In fact, each L0 segment contains lots of IDs for the deleted entities. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a way to accurately check the actual row count of the L0 segment? I suspect that there is no data in that segment. When performing a delete operation on non-existent data, the returned delete count value is 1. Does this cause empty L0 segments to be created? |
Beta Was this translation helpful? Give feedback.
-
You will see details of all segments, including the L0 segment. |
Beta Was this translation helpful? Give feedback.
-
|
Even delete non-existent data, the "entries" of L0 segment is not zero. |
Beta Was this translation helpful? Give feedback.
Answer selected by
pipi-olo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
upsert = delete + insert
Each time you upsert an entity, milvus will trigger a delete action to delete by the entity's id and an insert action to insert a new entity.
All the deleted IDs are records in L0 segment.
With more upsert/delete actions, you will see more L0 segments generated.
Datanode triggers compaction actions to erase entities by the IDs from L0 segment. The empty L0 segments mean their IDs have been processed. Empty L0 segments will be remove by GC(garbage collection) of milvus. GC action is triggered with an interval of 1 hour.
Intensive/continually upsert action will downgrade search performance because it is a heavier operation than normal insert.
The log doesn't show th…