Results Differ when I change the top_k + the answers to questions are wrong

[shahad2099]

Hello everyone,
Hope you're doing well !
I am trying to build a QA system . and after fine-tunning multilingual dpr on my dataset [Not English], and creating a pipeline
1- I noticed that the results are not trustworthy even when the question is simple :( especially when I upload a large number of documents [I use elastic search for indexing]
2- when I change top_k in retriever or reader the results changes , sometimes it gives better answers . why the behavior changes ?
3- do you have any suggestions so that my system will work better and be more accurate? and does the ES the best for DPR?

4-One last question what does score means here ?
<Answer {'answer': 'للرجبي', 'type': 'extractive', 'score': 0.834613025188446, 'context': 'مكان لنهائيات كأس العالم لكرة القدم 1998 الأخيرة، واستضافت كأس العالم للرجبي 2007 في نهائي أكتوبر 2007. فرنسا تستضيف أيضا السنوي سباق فرنسا للدراج', 'offsets_in_document': [{'start': 62606, 'end': 62612}], 'offsets_in_context': [{'start': 70, 'end': 76}], 'document_ids': ['dd4f545d403fa0ebeb0631ed5d17424e'], 'meta': {'name': 'wiki_08.txt'}}>,

Appreciate your help
Regards.

[bilgeyucel]

Hi @shahad2099, hope you're fine! It's nice to hear that you use Haystack to build a QA system🚀

1 - This might depend on many things. Did you check the model you use for your pipeline and its performance on the language you work with? Performance depends on top_k value if you're also working with large datasets.
2 - top_k value indicates the number of Documents to retrieve for Retriever and the number of Answers to collect from that Document list. You can check the more detailed answer in the documentation: Choosing the right top k values
3 - There are many things you can do. Play with the top_k values, try different models, combine sparse and dense Retrievers, change preprocessing on your documents... The best approach would be creating and evaluating different pipelines with a small dataset. Regarding DocumentStore, there is a page about choosing the right DocumentStore in the documentation.
4 - This score indicates the model's confidence in the given Answer. Read more about it here: Confidence Scores

Hope these help 🙌