Re-train roberta-base-squad2 QA model on domain data

[ashar1202]

For better results on domain specific data, how to re-train the roberta-base-squad2 QA model using MLM technique on domain data and then do fine tune on QA data for domain.?

[mkkuemmel]

Hi @ashar1202!

For the first step, I will point you towards huggingface's Github account. They have great examples on how to do language model training.

For fine-tuning on your own QA data, there is a great tutorial on Haystack's end.

Hope this was helpful! Have fun!

[ashar1202]

Thanks for the information, the idea iam trying is to train the model on domain data may be using masked data concept , so that the model will be aware of the domain and as we have very less data for training on QA domain specific it will be helpful to answer the question in better way as it understand the domain. still researching on it.

[parzival1l]

Hey, where you able to find a correct method for doing that? I am working on a similar task.

[ramshashaikh]

+1

[bogdankostic]

This talk by Nils Reimers might be interesting to you. He talks about how to incorporate new knowledge into language models.