Is it possible to add negatives to a retriever training?

[nicir]

Hello,

I am trying to train a Retriever with a custom language model and a custom dataset.

My dataset features about 60.818 positives, 83.356 negatives and 459.107 negatives I randomly picked from a wikipedia dataset (85% train, 15% test).

When I train my retriever it doesn't seem as the negatives would be used, though:

I also can't find num_negatives or something here:

So is it right to assume that negative examples are not effective and thus have been removed from training?

One more question: Is it right to let num_positives and num_hard_negatives 1 or should I change it according to the number of examples I have?

I would appreciate help a lot! Thank you!

[Zaker237]

Hallo,
are you using the DensePassageRetriever?
if yess you should prepare your data for the training the the following format:

{
    "dataset": str,
    "question": str,
    "answers": list of str
    "positive_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
    "negative_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
    "hard_negative_ctxs": list of dictionaries of format {'title': str, 'text': str, 'score': int, 'title_score': int, 'passage_id': str}
}

so your train and test data should be save as json file in that format which means that you should habe a json object for each query which directly include its possitive and negative documents.

you can get more details about it here

[nicir]

Yes, I am using the DPR and I have prepared my data the way you mentioned. It looks like this (as JSON of course):

DEMO for GitHub.txt

And I actually use code based on the tutorial you linked. That's why I'm wondering why the negatives are not used apparently.

[Zaker237]

to the second question depending on the number of positive and negative doc that you habe for each query you can change there parameters. i will suggest you to set num_positives to the number of positive doc and num_hard_negatives to the number of negative doc.

[nicir]

And when I remember right the hard negatives were retrieved by mistake by BM25 because it is somehow keyword-based.

That's why I built my dataset like that.

[TuanaCelik]

hard negatives are used to in short make the models life a bit harder while training :) They are passages that don't contain the answer to a query but appear as if it might because e.g. it has a lot of the relevant keywords. The num_hard_negatives is by default set to 1 for training meaning that we use just 1 example of a hard_negative in the batch, otherwise the batch sizes get quite high as there might be many hard negative examples.

You could increase this number of course, depends on whether bigger batches are ok in terms of capacity.
Hope that helps!

[nicir]

That helps a lot! I will play around with it a bit :)

But just to be clear: If I have a lot of hard_negatives in my array and set num_hard_negatives to 1, then the others will still be used but in another batch?

[TuanaCelik]

Exactly! At least this is what I also understand so I hope so :D I think we can make this clearer in documentation actually so thanks for opening this discussion. So yes, you may have many many hard negatives but imagine using all of them during a training round, that's a lot. So you can pick how many of them you use each time. My understanding is yes, if the same query is used in another batch, a different 'hard negative' could be selected this time around.

I'll make sure my understanding is right and post here again if anything else comes up.

[nicir]

great, thank you!