Need Help with Azure Search Vector Store as Retriever using semantic_hybrid_score_threshold

[Keith-Ho]

self.vector_store = AzureSearch(azure_search_endpoint=os.getenv("AZURE_SEARCH_SERVICE_NAME"), azure_search_key=os.getenv("AZURE_SEARCH_ADMIN_KEY"), index_name=self.index_name, embedding_function=self.embeddings.embed_query, semantic_configuration_name="my-semantic-config")
retriever = self.vector_store.as_retriever(search_type="semantic_hybrid_score_threshold",
                        k=50,
                        semantic_configuration_name="my-semantic-config",
                        score_type="score",
                        score_threshold=10.0
)

I currently have this code but I can't seem to get the scoring correctly. I couldn't find any examples online of anyone who used the semantic_hybrid_score_threshold so I'm not sure if my syntax is correct or not.

[dosubot[bot]]

Your syntax for implementing the semantic_hybrid_score_threshold in the AzureSearchVectorStoreRetriever seems mostly correct. However, ensure that you are setting the search_kwargs properly to include parameters like score_threshold, fetch_k, lambda_mult, and filter if needed. Here's an example setup:

retriever = self.vector_store.as_retriever(
    search_type="semantic_hybrid_score_threshold",
    k=50,
    search_kwargs={
        "score_threshold": 10.0,  # Ensure this is set correctly
        "fetch_k": 20,  # Example value, adjust as needed
        "lambda_mult": 0.5,  # Example value, adjust as needed
        "filter": "metadata_filter_expression"  # Example filter, adjust as needed
    },
    semantic_configuration_name="my-semantic-config"
)

Make sure that my-semantic-config is correctly set up in your Azure Cognitive Search service. The semantic_configuration_name is used to specify the semantic configuration for enhanced search capabilities ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[dosubot[bot]]

The issue you're experiencing is likely because the documents being retrieved do not meet the specified score_threshold of 0.10. In the semantic_hybrid_score_threshold search type, documents are filtered based on their scores, and only those with scores above the threshold are returned. If no documents meet or exceed this threshold, none will be returned ^[1].

To address this, you might want to verify the scoring mechanism and ensure that the scores being generated are indeed above the threshold you set. Additionally, check if the score_type is set correctly in your implementation, as it determines which score (e.g., score or reranker_score) is used for filtering. Adjusting the score_threshold or reviewing the scoring logic might help in retrieving the desired documents.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Keith-Ho]

@dosu

retriever = self.vector_store.as_retriever(search_type="semantic_hybrid_score_threshold",
                        k=50,
                        semantic_configuration_name="my-semantic-config",
                        search_kwargs={
                            'score_threshold': 0.1,
                        }
                    )

I noticed that inside my AI Azure Search Service, my documents have a @search.score value. Is it possible to add a filter in this code to remove any documents that have a @search.score = 1?

[dosubot[bot]]

To filter out documents with a @search.score of 1 in your Azure Search Vector Store setup, you can use the filter parameter within the search_kwargs. However, the specific syntax for filtering based on @search.score directly in Azure Search might not be straightforward, as the filter parameter typically applies to document fields rather than the score itself.

In the LangChain framework, filtering by score is usually handled by setting a score_threshold, but this doesn't allow for excluding specific scores like 1. If you need to exclude documents with a @search.score of exactly 1, you might need to implement a post-processing step after retrieving the documents to filter them out manually.

Here's a conceptual approach:

1. Retrieve the documents using your current setup.
2. Post-process the results to exclude documents with a @search.score of 1.

Example:

# Retrieve documents
retriever = self.vector_store.as_retriever(
    search_type="semantic_hybrid_score_threshold",
    k=50,
    semantic_configuration_name="my-semantic-config",
    search_kwargs={
        'score_threshold': 0.1,
    }
)

# Assume `results` is the list of retrieved documents with their scores
filtered_results = [
    doc for doc in results if doc['@search.score'] != 1
]

This approach allows you to filter out documents with a specific score after retrieval.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Keith-Ho]

@dosu

retrievers = []
if self.index_name == 'business':
    self.prompt = SYSTEM_PROMPT if custom_prompt == '' else custom_prompt + "{context}"
    common_space_index = ["csp-facets-prod-analysis", "csp-macess", "cns-resource-center"]
    for index in common_space_index:
        self.index_name = index
        if index == "csp-macess":
            self.vector_store = AzureSearch(azure_search_endpoint=os.getenv("AZURE_SEARCH_SERVICE_NAME"), azure_search_key=os.getenv("AZURE_SEARCH_ADMIN_KEY"), index_name=self.index_name, embedding_function=self.embeddings.embed_query, semantic_configuration_name="my-semantic-config")
            retriever = self.vector_store.as_retriever(search_type="semantic_hybrid_score_threshold",
                k=50,
                semantic_configuration_name="my-semantic-config",
                search_kwargs={
                    'score_threshold': 0.01,
                }
            )
        else:
            self.vector_store = AzureSearch(azure_search_endpoint=os.getenv("AZURE_SEARCH_SERVICE_NAME"), azure_search_key=os.getenv("AZURE_SEARCH_ADMIN_KEY"), index_name=self.index_name, embedding_function=self.embeddings.embed_query)
            retriever = self.vector_store.as_retriever(search_type="similarity_score_threshold",
                k=100,
                search_kwargs={'score_threshold': 0.83}
            )
        retrievers.append(retriever)
    retriever = EnsembleRetriever(retrievers=retrievers, weights=[1/len(retrievers)]*len(retrievers))
qa_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", self.prompt), MessagesPlaceholder("chat_history"), ("human", "{input}"),
    ]
)
question_answer_chain = create_stuff_documents_chain(self.llm, qa_prompt) 
rag_chain = create_retrieval_chain(
    retriever,
    question_answer_chain
)
result = rag_chain.invoke({"input":question, "chat_history":chat_history})

In my code, I can't see the documents until it gets returned by the result. Is it possible to filter the documents while they are being searched for instead of after the documents are already searched for?

[dosubot[bot]]

Yes, you can filter documents during the search process in Azure Search Vector Store by using the filter parameter in the search_kwargs. This allows you to specify conditions on document metadata fields to filter the documents as they are being searched. Here's how you can set it up:

retriever = self.vector_store.as_retriever(
    search_type="semantic_hybrid_score_threshold",
    k=50,
    semantic_configuration_name="my-semantic-config",
    search_kwargs={
        'score_threshold': 0.01,
        'filter': "metadata_field ne 'value_to_exclude'"  # Example filter condition
    }
)

This setup will apply the filter during the search process, ensuring that only documents meeting the specified criteria are retrieved. Adjust the filter expression to match your specific filtering needs based on the metadata fields available in your documents ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}