best pipeline for accuracy

[AhmadSawalhah]

Thank you for this great effort,
I have this project
user uploads docx file ( Django interface), I want to extract data from the uploaded file, such as
gender, country of origin, family members...etc.
I used an HF transformer and get about 40% accuracy of extracted data.
(How can I highlight the answer in frontend(Django))
I want to know what is the best pipeline to get the most accurate results, including everything
1- document store
2- the way of preprocessing the file ( size)
3- indexing
4- language model
5- retriever structure
6- reader
7..etc
performance is not important, I want best results
thanks
@hammer @icy @askainet @demarant

[tstadel]

Hey @AhmadSawalhah,

thanks for starting this discussion. :-)
I'm not too sure if I understood your use-case correctly. So please provide some additional information:

• What's the content of these documents like? Is it a description of a (single) person?
• How long in terms of words is a typical document?
• Which HF model did you try?
• When you say you got about 40% accuracy I assume you have some labeled data to evaluate on. How many labels do you have? You might want to use it as training data, too.
• Is the accuracy evenly distributed between the "classes" (gender, country, etc.)? Do some of them perform better/worse?

[AhmadSawalhah]

Hi @tstadel
1- my documents are interviews with refugees, It talks about one person, but she/he mentions other persons, like wife, children, ...etc
2- about 2500 - 3500 words
3- I used "bert-large-uncased-whole-word-masking-finetuned-squad"
4- no, I don't have labeled data, I just ask a question and find whether it is correct or not.
5- yes, some questions have more accuracy than others

[AhmadSawalhah]

@tstadel @javier @hammer @icy
any Ideas please!

[tstadel]

Hm, that's not easy to say, as we do not know which part (e.g. retriever or reader) is not performing well in terms of search quality. In general you might want to go for a basic QA-Pipeline with DPR-retriever and a SQuAD-trained retriever.
To give an answer to the question which component is not performing well, I'd recommend to create a small evaluation dataset (e.g. by using our Annotation Tool; about 50 questions should be sufficient for a first impression), export the data and feed it into your document store by calling add_eval_data(). Then you can call pipeline.eval() similarly to our Evaluation Tutorial. In the end, by using pipeline.print_eval_report() you should be able to determine which parts of your pipeline need improvements.

Depending on the complexity of your data you might also want to try named entity recognition models instead.