Combine FAQ QA with Extractive/Generative QA

[CDCapobianco]

Hi all,
I was wondering how to build a pipeline with FAQ QA that switches to Extractive/Generative if the FAQ QA score is not good enough.
I've only found this issue: #280 where they talk about it but the proposed solution is unclear to me:

For example, you could have two Finders: One that does extractive QA, one that finds most similar FAQs. At runtime, you could call both and combine the results. Depending on the use case, you could boost the results from FAQ a bit more as these might be "more trusted". Or: Only query the "slower reader", if FAQ doesn't return results above a certain score.

Querying the FAQ Pipeline, I've noticed that the score is quite unreliable. It's usually always between 0.53 and 0.58 even if I ask completely random questions and in general it doesn't seems to reflect the proposed answer quality so it would be quite complex to build a transparent system for the user where the slower reader is called only when FAQ provides a low-confidence score.

I've seen that in https://github.com/deepset-ai/COVID-QA they have used both FAQ and Extractive, so I'm wondering how they implemented it there.

Thanks in advance !

[julian-risch]

Hi @CDCapobianco if you already have an FAQPipeline, the next step could be to experiment with an ExtractiveQAPipeline. Both of them return Answer objects, which allows combining their results with a JoinAnswers node.

So you can build a single larger pipeline running both in parallel:

p_ensemble = Pipeline()
p_ensemble.add_node(component=retriever1, name="Retriever1", inputs=["Query"])
p_ensemble.add_node(component=retriever2, name="Retriever2", inputs=["Query"])
p_ensemble.add_node(component=Docs2Answers(), name="docs2answers", inputs=["Retriever1"])
p_ensemble.add_node(component=reader, name="Reader", inputs=["Retriever2"])
p_ensemble.add_node(
    component=JoinAnswers(join_mode="concatenate"), name="JoinAnswers", inputs=["Docs2Answers", "Reader"]

If you don't want to run them in parallel to speed up the pipeline, you could use a QueryClassifier. It's part of our tutorial on pipelines: https://haystack.deepset.ai/tutorials/11_pipelines

[CDCapobianco]

Ok so you suggest to use QueryClassifier to evaluate the question and decide which pipeline to execute.

Thank you