How to get the similarity between query and embedding when using the Retriever?

[Ma-Zijing]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# vector database
vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())

# The retriever
retriever = vectorstore.as_retriever(search_type="similarity", search_kwargs={"k": top_k})

# prompt input values
input_dict = {
        "hardware_components_list": static_dict["hardware_components_list"],
        "requested_permission_list": static_dict["requested_permission_list"],
        "app_components_list": static_dict["app_components_list"],
        "intent_filter_list": static_dict["intent_filter_list"],
        "restricted_api_list": static_dict["restricted_api_list"],
        "used_permissions_list": static_dict["used_permissions_list"],
        "suspicious_api_list": static_dict["suspicious_api_list"],
        "url_domain_list": static_dict["url_domain_list"]
    }

# prompt template
retriever_prompt = PromptTemplate.from_template(
        "Find the most similar app. The features are:"
        "hardware_components_list:{hardware_components_list}."
        "requested_permission_list:{requested_permission_list}."
        "app_components_list:{app_components_list}."
        "intent_filter_list:{intent_filter_list}."
        "restricted_api_list:{restricted_api_list}."
        "used_permissions_list:{used_permissions_list}."
        "suspicious_api_list:{suspicious_api_list}."
        "url_domain_list:{url_domain_list}.")

# input the content of the template to retriever to retrieve similar vectors in the database
response = retriever.invoke(retriever_prompt.invoke(input_dict).to_string())

Description

Hi! Now I want to implement an RAG approaches. So I have built a vector database and the retriever. I input the query to the retriever and invoke it, so the retriever will give me some contents that are related to my query. So my question is that how can I get the similarity between the query and the vectors so that I can observe how similar they are. Thank you!

System Info

No.

[dosubot[bot]]

To get the similarity scores between a query and the embeddings when using the Retriever in your RAG approach, you can use the similarity_search_with_score method provided by the Chroma class in the LangChain library. This method returns the documents most similar to the query along with their similarity scores.

Here is an example of how you can use this method:

# vector database
vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())

# The retriever
retriever = vectorstore.as_retriever(search_type="similarity", search_kwargs={"k": top_k})

# prompt input values
input_dict = {
    "hardware_components_list": static_dict["hardware_components_list"],
    "requested_permission_list": static_dict["requested_permission_list"],
    "app_components_list": static_dict["app_components_list"],
    "intent_filter_list": static_dict["intent_filter_list"],
    "restricted_api_list": static_dict["restricted_api_list"],
    "used_permissions_list": static_dict["used_permissions_list"],
    "suspicious_api_list": static_dict["suspicious_api_list"],
    "url_domain_list": static_dict["url_domain_list"]
}

# prompt template
retriever_prompt = PromptTemplate.from_template(
    "Find the most similar app. The features are:"
    "hardware_components_list:{hardware_components_list}."
    "requested_permission_list:{requested_permission_list}."
    "app_components_list:{app_components_list}."
    "intent_filter_list:{intent_filter_list}."
    "restricted_api_list:{restricted_api_list}."
    "used_permissions_list:{used_permissions_list}."
    "suspicious_api_list:{suspicious_api_list}."
    "url_domain_list:{url_domain_list}."
)

# input the content of the template to retriever to retrieve similar vectors in the database
query = retriever_prompt.invoke(input_dict).to_string()
results_with_scores = vectorstore.similarity_search_with_score(query, k=top_k)

# Print the results with their similarity scores
for doc, score in results_with_scores:
    print(f"Document: {doc}\nSimilarity Score: {score}\n")

In this example:

1. The similarity_search_with_score method is used to retrieve the documents along with their similarity scores.
2. The method returns a list of tuples, where each tuple contains a document and its corresponding similarity score.

This will allow you to observe how similar the retrieved documents are to your query ^[1].

To continue talking to Dosu, mention @dosu.

[anudevmanjusatheesh]

But can we add arguments like search_type which should be "mmr" for my use case. With the similarity_search_with_score that are not acceptable arguments I suppose and with as_retriever I can use these things. How can I deal with it.