Predicting answers using deberta-v2-xlarge-cuad finetuned on CUAD dataset

[amitarjun]

I'm trying to use the haystack pipeline to extract the answers using deberta-v2-xlarge-cuad fine-tuned on the CUAD dataset.
https://github.com/TheAtticusProject/cuad

However, most of the time, the model makes junk predictions. Is there a way to implement It correctly?

[julian-risch]

Hi @amitarjun On what data does the model make "junk predictions"? One way you could check that the model was loaded correctly and works correctly is to apply it to the CUAD dataset it was trained on (and tested on) and compare the performance on that data to the numbers reported in the paper https://arxiv.org/pdf/2103.06268.pdf

If you could share some code as a google colab notebook I'd be happy to look deeper into it. 🙂

[descentis]

Hi @julian-risch thanks for your reply.
I was actually checking the predictions on CUAD dataset. Surprisingly, passing the clean_function parameter during the file_to_doc conversion removes all the issues I was facing. However, for some reason the same prediction is not working with the sreamlit app. I'm attaching the streamlit repo for your reference

[tstadel]

Hey @descentis,
I'm happy to help. What do you exactly mean with "prediction is not working"?

• If there's an error: Please provide the stacktrace, better yet open an issue.
• If there are still junk predictions: Please provide the working code so I'm able to figure out what causes the differences.

Just out of curiosity: Why did you remove the clean_function in your latest commit? Isn't that what solved the discrepancies in the first place?

[descentis]

Hi @tstadel
The prediction works well on the colab notebook (even without GPU instance). However, it doesn't work with the streamlit server. Unfortunately, there is no error, it just keep getting stuck at the prediction stage.

[amitarjun]

Hi @julian-risch
I was able to work around the issue. For some reason, the inference is getting stuck because of the multiprocessing. While loading the model using the FARMreader, switching off the multiprocessing seems to work. Similar is the case for training/fine-tuning the model. The num_processes should be set to 1 in the reader.train() function.

Any comments?

[julian-risch]

The tokenizers parallelism in the transformers library used under the hood by Haystack and the multiprocessing that Haystack uses internally might have cause a deadlock in your case for some reason. Setting num_processes to 1 is probably the best option for now. An alternative that you could try out is to set the environment variable TOKENIZERS_PARALLELISM to False.