Tutorial 9: One dataset for test & dev

[nicir]

Hello,

in Tutorial 9 when training a retriever with data from facebook-ai there is only one dataset for testing and evaluation. Is there a reason why?

They are defined as:

• train_filename: training filename
• dev_filename: development set filename, file to be used by model in eval step of training
• test_filename: test set filename, file to be used by model in test step after training

[mayankjobanputra]

Hi @nicir,

I think it is a common practice to get validation accuracy.

In most real-world applications/competitions, test data doesn't come with labels and it is just used for generating model inferences that would be evaluated later by humans or automatically in case of competitions. Dev data is anyway not used for loss computation, so there's no harm in using it for testing the model's accuracy. Although, it should be the same as dev/validation accuracy.

Does this answer your question? Please let me know if I misunderstood anything :)

[nicir]

Yeah, thank you! That answered my question. I was just wondering about it. I'll just leave it that way.