Are the results returned by hybrid search guaranteed to be in order from most to least relevant?

[bjeffrey92]

I have been making the assumption that the results of hybrid search will be returned in order from most to least similar to the query, because this would seem to be the most logical way to order the data. However, as far as I am aware, this is not documented anywhere.

Furthermore, can I ask what the meaning of the "distance" field in the returned data is? Is this truly a distance, where a larger value would imply a worse match to the query, or is it a similarity score where a larger value would imply a closer match to the query?

I am using the reciprocal rank fusion ranker with hybrid search, so I was expecting that the value which is returned with the data would be the output of this function. In which case it would be a score, where larger values mean a better match, but then naming the metric "distance" in the output is quite confusing

Thank you in advance for your help 👍

[yhmo]

Search results are always ordered from most to least relevant, both for search() and hybrid_search().
In a search result set, you will see an id-distance or id-score pair for each item. No matter whether we call it "distance" or "score", it is a number output from a mathematical algorithm. The mathematical algorithm calculates the similarity/distance of two embeddings.
All the mathematical algorithms are listed here: https://milvus.io/docs/metric.md

• L2 - Euclidean distance
  The distance range is [0, ∞), 0 is the most relevant, ∞ is the least relevant. So, if you choose L2 metric type, the search result is ordered by distance in ascending order.

• IP - Inner product
  The similarity range is [-1, 1], for normalized embeddings, 1 is the most relevant, -1 is the least relevant. So, if you choose IP metric type, the search result is ordered by distance in descending order.

• Cosine
  The similarity range is [-1, 1], 1 is the most relevant, -1 is the least relevant. So, if you choose IP metric type, the search result is ordered by distance in descending order.

For hybrid_search(), each hybrid search request is combined by several sub-requests, each sub-request is executed separately. So, the first step of hybrid_search() is get N search result sets from N sub-requests. The second step is to merge the N search result sets into one final result set.
Since different sub-requests might use different metric types, different metric types have different ranges, all the distance/score values of the N result sets are mapped to the same range [0, 1] by some "activate functions".
The activate functions are listed in this table:

Now all the distance/score values are in the same range [0, 1] , 1 is the most relevant, 0 is the least relevant. The next step is: merge the N result sets into one final result set.
Currently, milvus supports two rerankers for hybrid_search(): RRFRanker and WeightedRanker. The algorithm of rerankers is listed here: https://milvus.io/docs/reranking.md

After rerank, the final result set is ordered in descending order.

[bjeffrey92]

OK thanks for clarifying, that is very helpful. Although I might suggest that it would be helpful to rename that field in the hybrid search output to score rather than distance?

[yhmo]

Changing the field name might affect the legacy client-side script of users.
For example:

for res in results:
  print(res["id"], res["distance"])

If we change the name, the legacy script might run into trouble.