Embeddings not showing up in Milvus distance searches right after insertion. How do you deal with this?

[HikmetOzdemir]

I'm running a high-throughput pipeline that is inserting hundreds of embeddings per second into Milvus. I use a "search before insert" strategy to prevent duplicates and close-embeddings, as they are not useful for my use case. However, I’m noticing that many recently inserted embeddings aren’t searchable immediately, which leads to duplicate entries getting in.

I understand Milvus has an eventual consistency model and recently inserted data may not be visible until segments are flushed/sealed, but I was wondering:

• How do you handle this kind of real-time deduplication?
• Do you manually flush after every batch? If so, how often?
• Has anyone implemented a temporary in-memory dedup buffer or shadow FAISS index to work around this?
• Any official best practice for insert + dedup pipelines in high-throughput scenarios?

Any insight would be appreciated.

[xiaofan-luan]

Thanks for raising this — it’s a very relevant topic, and we’ve seen similar needs from other users recently.

Could you briefly share more context on your use case and why deduplication at the vector level is essential? We've received several requests around this, but we're still building a complete picture of the underlying motivations. We're actively evaluating ways to support this natively in Milvus.

Using a strong consistency setup and performing a search before insert can help reduce duplicates. That said, with ANN-based search, it's important to note that deduplication isn't guaranteed to be perfect due to the approximate nature of the algorithm.

You don’t need to flush after each batch — newly inserted data is immediately visible for search. In fact, we have introduced growing indexing, which enables IVF-based indexes to support real-time inserts without waiting for a segment flush. This significantly reduces latency for high-throughput pipelines and improves search consistency.

Let us know if you're open to discussing further — we'd love to better understand your requirements and explore how we can help.

@czs007
please take a deep look into it

[HikmetOzdemir]

Thanks for the reply. I’m a developer at a small company building computer vision products.

We’re using facial embeddings for recognition and tracking, and embeddings that are too close (below a certain L2 threshold) are essentially redundant. Filtering them out is crucial. It helps us avoid storing near-duplicates and significantly reduces the index size in RAM, which in turn increases how many days of footage we can retain for similarity search. For scale, we’re using a DiskANN index to maximize embedding capacity in the database. Currently, thanks to Milvus, we are comfortably storing ~30 million embeddings with their metadata. Our expectation is that we should be able to store ~100 million embeddings with our hardware. This is per-server.

A simple example: imagine a person standing in front of a camera for 30 seconds without moving. At 25 FPS, that’s 750 facial embeddings of the same person. This is unnecessary. It bloats the database and hurts longevity. There are more complex scenarios than this, so we’d like to deduplicate as part of insertion, not via a tracking algorithm.

On a single device, our current throughput is 800 FPS at 1080p. This includes detection inference, embedding generation, metadata collection, and writing to both Milvus and MySQL.

From my observations, at this throughput, without manually flushing the collection (which halts performance), an unacceptable number of duplicates make their way into the collection. I believe this is due to the high throughput and batch inserts (we are running deduplication in the batch itself as well).

You said: “You don’t need to flush after each batch — newly inserted data is immediately visible for search.” But unfortunately this does not seem to be the case for us. Deduplication via a similarity search does work. We are seeing a reduction in size of about 70%. However, in a relatively small collection of ~2M embeddings, there are still a concerning number of embeddings with “duplicate-considered” distances. I could get you some actual numbers, but from my observation, there seems to be anywhere from 2-10 “duplicate-considered” embeddings per embedding. Some of their L2 distances are less than 0.00…

Thank you all so much for the work you do at Zilliz ❤️

[liliu-z]

Thanks for the reply. I’m a developer at a small company building computer vision products.

We’re using facial embeddings for recognition and tracking, and embeddings that are too close (below a certain L2 threshold) are essentially redundant. Filtering them out is crucial. It helps us avoid storing near-duplicates and significantly reduces the index size in RAM, which in turn increases how many days of footage we can retain for similarity search. For scale, we’re using a DiskANN index to maximize embedding capacity in the database. Currently, thanks to Milvus, we are comfortably storing ~30 million embeddings with their metadata. Our expectation is that we should be able to store ~100 million embeddings with our hardware. This is per-server.

A simple example: imagine a person standing in front of a camera for 30 seconds without moving. At 25 FPS, that’s 750 facial embeddings of the same person. This is unnecessary. It bloats the database and hurts longevity. There are more complex scenarios than this, so we’d like to deduplicate as part of insertion, not via a tracking algorithm.

On a single device, our current throughput is 800 FPS at 1080p. This includes detection inference, embedding generation, metadata collection, and writing to both Milvus and MySQL.

From my observations, at this throughput, without manually flushing the collection (which halts performance), an unacceptable number of duplicates make their way into the collection. I believe this is due to the high throughput and batch inserts (we are running deduplication in the batch itself as well).

You said: “You don’t need to flush after each batch — newly inserted data is immediately visible for search.” But unfortunately this does not seem to be the case for us. Deduplication via a similarity search does work. We are seeing a reduction in size of about 70%. However, in a relatively small collection of ~2M embeddings, there are still a concerning number of embeddings with “duplicate-considered” distances. I could get you some actual numbers, but from my observation, there seems to be anywhere from 2-10 “duplicate-considered” embeddings per embedding. Some of their L2 distances are less than 0.00…

Thank you all so much for the work you do at Zilliz ❤️

Thanks for this sharing, Did you get a chance to try Strong consistency for the search check before insertion? This might add hundreds of milliseconds to the latency for current implementation, but we are working on optimize it.

[xiaofan-luan]

Thanks for the reply. I’m a developer at a small company building computer vision products.

We’re using facial embeddings for recognition and tracking, and embeddings that are too close (below a certain L2 threshold) are essentially redundant. Filtering them out is crucial. It helps us avoid storing near-duplicates and significantly reduces the index size in RAM, which in turn increases how many days of footage we can retain for similarity search. For scale, we’re using a DiskANN index to maximize embedding capacity in the database. Currently, thanks to Milvus, we are comfortably storing ~30 million embeddings with their metadata. Our expectation is that we should be able to store ~100 million embeddings with our hardware. This is per-server.

A simple example: imagine a person standing in front of a camera for 30 seconds without moving. At 25 FPS, that’s 750 facial embeddings of the same person. This is unnecessary. It bloats the database and hurts longevity. There are more complex scenarios than this, so we’d like to deduplicate as part of insertion, not via a tracking algorithm.

On a single device, our current throughput is 800 FPS at 1080p. This includes detection inference, embedding generation, metadata collection, and writing to both Milvus and MySQL.

From my observations, at this throughput, without manually flushing the collection (which halts performance), an unacceptable number of duplicates make their way into the collection. I believe this is due to the high throughput and batch inserts (we are running deduplication in the batch itself as well).

You said: “You don’t need to flush after each batch — newly inserted data is immediately visible for search.” But unfortunately this does not seem to be the case for us. Deduplication via a similarity search does work. We are seeing a reduction in size of about 70%. However, in a relatively small collection of ~2M embeddings, there are still a concerning number of embeddings with “duplicate-considered” distances. I could get you some actual numbers, but from my observation, there seems to be anywhere from 2-10 “duplicate-considered” embeddings per embedding. Some of their L2 distances are less than 0.00…

Thank you all so much for the work you do at Zilliz ❤️

Is there a way you can batch a insert of the embeddings from last 1 minutes or 10 minutes and do a mini clustering on the data?

So each each window you should ingest several unique images

[xiaofan-luan]

After fix primary key dedup, this vector dedup seems to be a very interesting topic to work with.
We can definitely do some experiment and see it's working