Retriever Scores

[nachoperezzv]

Hello everyone.

I have a question about the scores that the retriever models give you as an output. It's not clear for me what that means exactly. For example. anytime I ask a question which answer isn't in any document of document store it gives me back a score that is higher than the times when it hits the right document. By the way I'm talking about DensePassageRetriever. And I'm using this set of models:

• IIC/dpr-spanish-question_encoder-squades-base
• IIC/dpr-spanish-passage_encoder-squades-base

If any of you can help would be great. Thanks a lot.

[mayankjobanputra]

Hey @nachoperezzv,

Have you trained/fine-tuned these models using your data? It is possible that the embedding these models are generating is not suitable for your own data.

[nachoperezzv]

These are fine-tuned models from hugging face, but they haven't been fine-tuned by me. I've tried with another type of Dense Retriever such as the Embedding, with sentence-transformers models. It still gives me "random" scores. Any clues, besides the above, why this might be happening ?

Thanks a lot for your answer!

[mayankjobanputra]

In general, the models do require some sort of fine-tuning on your own data, even if they are already fine-tuned because your data might be very different from what the model is fine-tuned on.

Additionally, what's the frequency of randomness? Do you have any guesses or have analyzed how often the scores are wrong? Are they wrong for a specific topic or set of topics?

Is it possible that your encoded documents are very large and you're not performing any kind of splitting before encoding/indexing? This can mess up the retrieval process.

[nachoperezzv]

I'm not splitting the documents. In fact, some of them are long, maybe there're some that have more than 1000 tokens/per document. So I might try the splitting. Thanks a lot!