Is it possible to store a bitset in metadata and retrieve it?

[jclab-joseph]

The docs describe bitset/bitmap.

• https://milvus.io/docs/bitset.md
• https://milvus.io/docs/bitmap.md

But I don't know how to use it.

Is there a way to add a bitset like binary_vector to the metadata field and then search it with AND operation?

I want to implement a bloom filter by adding a bitset to metadata.

[xiaofan-luan]

I think the goal is to bypass the external filtering result with a bloom filter?

Something like a large IN operation but you can not pass a very long list(like more than 1M IDs) that's why you are looking for a work around?

Can you tell more details about the use case?

[jclab-joseph]

@xiaofan-luan Yes, I want to implement a branch like git in a situation where there are more than 1M documents.
To do this, I came up with a way to hash all the branch names that the document belongs to, store them as a bloom filter, and then query them.

[xiaofan-luan]

@xiaofan-luan Yes, I want to implement a branch like git in a situation where there are more than 1M documents. To do this, I came up with a way to hash all the branch names that the document belongs to, store them as a bloom filter, and then query them.

just want to be clear, you only want to filter on a small set of document ? if it's less than 1M, can you just do an IN operation

[xiaofan-luan]

@tedxu

please take an eye on this issue

[jclab-joseph]

can save all document IDs in metadata DB. It seems possible with in-operation for small number.
But in my condition, there are more than 1M documents in the branch, and you can create a new branch after adding data.

Branch A (count: 1M) ─────► Branch B (1.2M) ─────► Branch C (1.3M)
                      │
                      └───► Branch D (1.1M)  ─────► Branch E (1.4M)

[xiaofan-luan]

guess you are looking for snapshot feature?

[xiaofan-luan]

each of the branch is simply a snapshot of the collection

[xiaofan-luan]

I have plan to implement this feature in the next 2-3 months

[jclab-joseph]

Yes, I think the snapshot feature is correct!
That's great news! I'm looking forward to it! :)

[jclab-joseph]

Out of curiosity... is there any way to implement this within the current available means?

[xiaofan-luan]

we have several user try to implement it with collection backup and restore.
It is slow but could be an alternative.

[jclab-joseph]

My need is to be able to search within different snapshots instantly. Would this be possible if snapshots were implemented in milvus?

[xiaofan-luan]

then this is more like a time travel feature.

We used to work it for now but for it's kind like hard to maintain and might take many resources that's why they decided not to keep this feature.

[xiaofan-luan]

to make everything able to search, you will need to load all the historical data into memory and that might be too many.

We are gonna to offer time travel feature probably Zilliz only through vector lake solution in the next quarter of year.