BM25 full-text search is not working; it only supports exact queries?

[262146150]

I used Milvus's full-text search, created the schema and BM25 function as required, and after successfully inserting the data, I found that during retrieval the full-text search did not take effect—it only returns results when an exact query is entered. I wonder what the issue might be?

[yhmo]

Set analyzer_params.type to "chinese". Read this doc for more info: https://milvus.io/docs/analyzer-overview.md

import random
import time

from pymilvus import (
MilvusClient,DataType,Function,FunctionType, CollectionSchema,FieldSchema
)

client = MilvusClient(uri="http://localhost:19530")

collection_name = "AAA"
client.drop_collection(collection_name=collection_name)

functions = [
    Function(
        name="bm25",
        function_type=FunctionType.BM25,
        input_field_names=["text"],
        output_field_names="sparse_vector",
    )
]

analyzer_params = {
    "type": "chinese",
}
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=512, enable_analyzer=True,
                analyzer_params=analyzer_params),
    FieldSchema(name="sparse_vector", dtype=DataType.SPARSE_FLOAT_VECTOR),
]

schema = CollectionSchema(fields, "", functions=functions)
client.create_collection(
    collection_name=collection_name,
    schema=schema,
)
index_params = client.prepare_index_params()
index_params.add_index(field_name="sparse_vector", index_type="SPARSE_INVERTED_INDEX", metric_type="BM25")
client.create_index(collection_name, index_params)
client.load_collection(collection_name=collection_name)
print(collection_name, "created")


client.insert(collection_name=collection_name, data={"id":1, "text": "我是个客服"})
client.insert(collection_name=collection_name, data={"id":2, "text": "搜索互联网"})
client.insert(collection_name=collection_name, data={"id":3, "text": "系统设置"})
client.insert(collection_name=collection_name, data={"id":4, "text": "我命由我不由天"})
client.insert(collection_name=collection_name, data={"id":5, "text": "客服搜索设置"})
client.insert(collection_name=collection_name, data={"id":5, "text": "肯定搜不到这条"})
client.flush(collection_name=collection_name)
print("data inserted")

res = client.search(collection_name=collection_name,
                    data=["客服搜索设置"],
                    limit=10,
                    anns_field="sparse_vector",
                    output_fields=["text"]
                    )
for r in res:
    for k in r:
        print(k)

Outputs:

AAA created
data inserted
{'id': 5, 'distance': 3.390374183654785, 'entity': {'text': '客服搜索设置'}}
{'id': 3, 'distance': 1.2800674438476562, 'entity': {'text': '系统设置'}}
{'id': 2, 'distance': 1.011626124382019, 'entity': {'text': '搜索互联网'}}
{'id': 1, 'distance': 1.011626124382019, 'entity': {'text': '我是个客服'}}

[262146150]

Thank you very much. It turns out that the error was due to incorrect analyzer_params settings. Additionally, could you briefly explain the meaning of 'distance' in full-text search and in hybrid search? Or is there any corresponding documentation available?

[yhmo]

With BM25 function, milvus generates a sparse vector internally for each text, and uses the sparse vector to determine the document's relevance to a given query and ranks documents based on their relevance scores. You can search "bm25 algorithm" on www to get more info. It is not an invention or milvus.