Evaluation on DensePassageRetriver

[Zaker237]

Hi Dear @ALL
ich used the DensePassageRetrieval from haystack to train my own DPR model from deepset/gbert-base. the training were successfull and i did some evaluation with GermanDPR_eval data after fine-tuning of my nodel with GermanDPR. By the evaluation i get the same value for Map and Mrr and ich was wondering. still yet i'm not able to understand why the mrr and the map are egal for each value of k. do someone have an idea or an explaination to that??
thank you

[TuanaCelik]

Hey @Zaker237
MAP and MRR are very similar metrics. But not quite.
MRR takes into account only the highest-ranked positive sample (even if there were multiple positive samples)
MAP takes into account how high ALL the relevant samples are ranked.
But, if you have only one positive sample/document per query in your dataset, the two metrics effectively become the same. Because there are no 'other' relevant documents/samples that MAP can take into consideration. I believe that this is the case with the GermanDPR dataset. This is why you're getting the same results.

This post here does a good job of explaining the two.

[Zaker237]

Thank you verry much @TuanaCelik.
it is now clear. i did think on the fact that GermanDPR just have one positive document.