How should I synchronize milvus with postgresql or another database?

[amano-kenji]

Let's assume that postgresql contains e-commerce items. For each e-commerce item, milvus contains vector embeddings.

When you delete an e-commerce item from postgresql, you want to delete all vector embeddings associated with the item.

It should normally work, but it can sometimes fail. Thus, garbage vector embeddings can remain.

Can I prevent garbage from accumulating in milvus in the first place? If I can't, is there an efficient garbage collection algorithm? I don't want to do periodic full scan on postgresql and milvus.

[xiaofan-luan]

VTS https://github.com/zilliztech/vts
is a tool we build to sync data from multiple different place to milvus (including milvus to milvus)

I guess your requirement a subscribe on PG logs and replayed in milvus.

Please submit an issue in vts repo and feel free to contribute if you are interetest