Document search from faiss based on Documents score.

[VikRathod]

I developed a chat app using faiss document store to find similar documents for a query. It retrieves top 5 docs, with the highest as "High_score" and the lowest as "minimum score." A condition is applied, requiring values to meet specific criteria (e.g., minimum_value_1 >= 0 and < 1.04, maximum_value_1 <= 1.45).
The application uses a condition to assess query correctness and retrieve relevant documents accordingly. Unfortunately, it classifies some relevant queries, leading to a failure in providing answers for queries that should be considered relevant.

Is there any solution that you guys have, or i am missing something in the process.

[anakin87]

Hello, @VikRathod!

I found it a bit difficult to understand your use case and your issues.
Could you try to explain better and/or provide a reproducible example?

[VikRathod]

Hi @anakin87,

I have created an application similar to ChatPDF.
In this app, I trained a retriever model using closed domain PDFs. The purpose is to ask questions and receive relevant documents based on the queries. However, I encountered an issue where the model retrieves documents from the Docstore even for irrelevant queries. For instance, when I ask unrelated questions such as "who is Vikas Rathod?" or "what is the weather today?", the model still searches for documents. What I want is for the model to stop searching for documents from the embedding Docstore when it encounters irrelevant queries.

To address this, I experimented with different threshold values using both relevant and irrelevant queries, but the approach failed to work effectively for relevant queries as well.

[anakin87]

Are you performing Semantic Search (input: query; output: Documents)
or Question Answering (input: query; output: Answers)?
If Question Answering, is it Extractive or Generative?

[anakin87]

(If you are performing Generative Question Answering, also this discussion can be useful.)

[VikRathod]

Thank you @anakin87,
Yes i am performing generative Question Answering using text-davinci-003 model on retrieved relevant documents based on query. my concern is when user ask irrelevant question then that time , I don't want waste my API key usage so for that what i can do?

[anakin87]

I understand your point.
But if your scores are not very reliable, using this prompt can help you.

In any case, I would suggest you try gpt-3.5-turbo instead of davinci (with the PromptNode)
because it is cheaper but might work well for your use case...