QA strategies for a chat, not single one shot questions.

[Uranium2]

Hello,

They are more and more examples online about QA over documents. Made mine with FAISS over PDF/CSV and it works well enough for single questions.

But in a real conversation, the "chat" should remember old questions and history, so I gave memory to my Chat. Great now the GPT has access to history and can use it has context. BUT, I did not find a way of having a natural conversation with the GPT understanding that the last given question might be related to an old question.

For example, let's have an imaginary PDF with 1 page = 1 vector. Not great, not bad. Anyway, the content of the PDF is the structure of a company or a family, so people have roles/titles and links to other people. |

If the user asks "Who is X?"
I vectorise the question "Who is X?" (I also remove stop words and punctuation in the question and pdf for better vectorisation similarity)
The GPT answers something like: "X is the responsible for the industry department"
And I retrieve sources so the user can check the PDF with the page.

All this example is "simple" QA.

What if I ask a new question:

"Who he is working with?", if I keep my simple logic of vectorisation, I will not find any relative document and the GPT will say "I don't have enough information to answer this"

So I imagined 2 possible strategies.

The first naive one would the concact all questions history with the new question to have a larger vector for research. I think it could work, but if the user ask inbetween a question that has no connection to anything, it will make the question vector less relevant to documents. The distance will inscrease.

My second idea was to use GPT to use the History chat to rewrite the question with the history if he finds a link between the questions.

Let's have an imaginary example from this prompt:

---------- INSTRUCTION ----------
From this HISTORY and a QUESTION, rewrite the QUESTION with HISTORY if you find a link between them. If you don't know, just write the original question.
---------- INSTRUCTION ----------

---------- HISTORY ----------
Humain: Who is X?
AI: X is the director of ABC
Humain: Is Y member of the team BCD?
IA:  Y is not a member of BCD, but he works with the team EFG.
Humain: Can you list me the members of EFG team?
AI: Members are : Z, I and P
---------- HISTORY ----------

QUESTION: Who are the coworkers of the first guy?

Do not answer to the question, follow the instructions.

Who are the coworker of X, the director of ABC?

Do you think this strategy would be viable for a smoother chat experience? What I don't like about this, is that instead of doing:
Vector -> document retrieve -> prompt -> gpt -> answer
I do:
Prompt 1 -> gpt -> new question/original question -> vector -> document retrieve -> promp2 -> gpt -> answer.

So it will make 2 GPT calls, so double the money, even if the first GPT call should use less tokens.

Any thoughts, or criticism? What would you do?

[nilsdacke]

Yes, rewriting the chat history into a standalone question is probably your best bet. LangChain supports this in a manner similar to what you propose.

They use a prompt similar to this:

"""Given the following conversation and a follow up question, rephrase the follow up question to be """
"""a standalone question, in its original language.

Chat history:
{chat_history}
Follow up question: {question}
Standalone question:"""

I am doing the same thing. I think this works well. The main problem I have seen is that sometimes the rewritten query makes the wording of the response not fit naturally to the original question.

[Uranium2]

I've made several tests, and the best was removing answers from the chat history, so that the gpt does not retake works he generated.
I also asked in the prompt write the original {question} if no link was found in the questions. With this method I mostly don't rephrase the question too much. For example it will keep abbreviations like the original question was.

---------- INSTRUCTION ----------
From this HISTORY and a QUESTION, combine the QUESTION with HISTORY if you find a link between them. If you don't know, just write exactly "{question}".
---------- INSTRUCTION ----------

---------- HISTORY ----------
Who is X?
Is Y member of the team BCD?
Can you list me the members of EFG team?
---------- HISTORY ----------

QUESTION: Who are the coworkers of the first guy?

Do not answer to the question, follow the instructions.

I am sure this method has weakness, because you don't control the which history question might be related to original question. At the moment it seems to works pretty well. And also the cost is marginal since the number of token are low.