Filtering Vector Similarity Search

[szuwgh]

The article says that the vector data is filtered out according to 0
and 1 and loaded into memory together for similarity calculation.
Does the filtered vector use faiss to build an index in real time and then query it? Or you built your own vector
indexing module. Vectors can be filtered in the query。

[yhmo]

Index is built by Faiss, but we add some new logic in the search algorithm, a bitset array is input to skip some vectors.

For example, there are 10000 vectors, we use Faiss to build an index for the 10000 vectors. User search with an expression "xxx > 500", then we construct a bitset array, the array has 10000 elements, some elements are 0, others are 1. Then we input the bitset array along with the index to the search engine, the search engine will use the bitset array to skip some vectors.

[szuwgh]

Thank you for your answer. If there are 10 vectors, use faiss to build an index, then input a vector to query top 5, and output [5,2,6,3,1]. I need to use scalar filtering, such as xxx>10, output bitset [0,1,0,1,1,0,1,0,0,1], and then take the intersection to output [5,3,1]. But our results need top 5, and only three results are output. This method may lose some results

[yhmo]

Right, the current logic cannot ensure you can get topk results.
In common cases, users have tens of thousands of vectors, and even millions of vectors. Most of the time, the result count is larger than topk.

[zaytay]

Index is built by Faiss, but we add some new logic in the search algorithm, a bitset array is input to skip some vectors.
hello，can you please describe the detail of “some new logic in the search algorith“ or link some article related to this。The Faiss index is immutable，want to know the bitset array how work with the underlying index，thx

[yhmo]

hello，can you please describe the detail of “some new logic in the search algorith“ or link some article related to this。The Faiss index is immutable，want to know the bitset array how work with the underlying index，thx

Milvus use a middleware(we called "knowhere") to wrapper the index libs including Faiss, it is also open-source. You can browse the source code to learn.
The index interface is defined at here:
https://github.com/milvus-io/knowhere/blob/main/knowhere/index/VecIndex.h

We can see the query() interface has a parameter "bitset":
Query(const DatasetPtr& dataset, const Config& config, const faiss::BitsetView bitset)

For more information, you can go deep into the source code.

[zaytay]

hello，can you please describe the detail of “some new logic in the search algorith“ or link some article related to this。The Faiss index is immutable，want to know the bitset array how work with the underlying index，thx

Milvus use a middleware(we called "knowhere") to wrapper the index libs including Faiss, it is also open-source. You can browse the source code to learn. The index interface is defined at here: https://github.com/milvus-io/knowhere/blob/main/knowhere/index/VecIndex.h

We can see the query() interface has a parameter "bitset": Query(const DatasetPtr& dataset, const Config& config, const faiss::BitsetView bitset)

For more information, you can go deep into the source code.

That is very helpful， thx