Best Approach for Natural Language Inferencing

[ss2342]

Hello all! I am trying to use haystack to integrate my fine-tuned transformer model with my training set. The way that I would like to use Haystack's functionality is that I want to feed it a number of documents and I want the documents to be correctly segmented into paragraphs, in the same way that they are in the document. I would then send in to my model a query and a paragraph which should get me a YES or NO answer. What would be best approach to do this with Haystack?

The part that I am struggling with is using Haystack to break up a document into paragraphs. I am not sure what would be best way to go about that.

[ZanSara]

Hey @ssingh1997 have you tried to use the PreProcessor? It's designed specifically for this task, among others.

[ZanSara]

Try with split_by='word' plus respect_sentence_boundary=True. It's unintuitive, but it works better than split_by="passage"

[ss2342]

Thank you once again! I just tried this out and it gave me a much better output. I was just curious though how we can use the PreProcessor to adhere to the paragraph sizes in the document itself. For example, if one paragraph has a sequence length of 100, and the next one is 300, is there any way to extract each paragraph as its own entity? I checked the PreProcessor code and saw that if we do not pass in the split_length size, it will always split every 200 words. By having a fixed split_length, some text from the next paragraph gets incorrectly included in the paragraph prior.

[ZanSara]

Good question... I think right now there's no such functionality in PreProcessor, also due to the fact that it's not always 100% clear what represents a "paragraph" in ASCII/UTF8. Will every \n define a paragraph? Only \n\n? How about \f? It's totally solvable, we just never came to think about this 😄

If you need this feature, feel free to contribute it! We always welcome PRs, especially small and well-defined ones like this. Feel free to tag me so I can review it and speed up the process a bit.

[ss2342]

sounds good, I can create a new issue and start working on it! I will make sure to tag you on it as well.

[TuanaCelik]

Hey @ssingh1997 - I marked this question as answered for now assuming you created the issue. Feel free to post a link to it here so that anyone who finds this thread can follow along 👍🏽