Wrong borehole and metadata extraction, verb-based approach

[letao]

Continuing the investigations from #198 and #222, we should look at a way to better identify false positive boreholes (document pages that contain similar material descriptions but that are not themselves boreholes). For example: 35613_22.pdf

In this issue we should look into absence of verbs in the text as indication that the document in question is a borehole.

As an example we should aim to eliminate similar false positives to the file: 45004_8.pdf:

Pages where data is extracted but aren't actual boreprofiles are available on the s3 bucket under false_positive_boreprofiles

[lhaibach]

False Positive Boreprofiles (tickmarks mean file was wrongly detected as a boreprofile)

Filename	Baseline in Assets	your approach
Berichte_NAB_08-021_289.pdf	✔️	✔️
Berichte_NAB_08-025_EWS_8.pdf	✔️	✔️
24361_8.pdf	✔️	✔️
1677_62.pdf	✔️
4828_4.pdf	✔️
1062_7.pdf		✔️
1630_83.pdf		✔️
1677_11.pdf		✔️
1801_8.pdf		✔️
24131_28.pdf		✔️
29192_19.pdf		✔️
43283_12.pdf		✔️
44159_44.pdf		✔️
45004_7.pdf		✔️
45004_8.pdf		✔️
Berichte_NAB_08-025_EWS_20.pdf		✔️

Most of the pages that do not get identified as boreprofiles by the baseline classifier in the assets dataextraction repository, get classified as text (where we check for text density within textblocks) or do not pass the boreprofile checks (basically checks ratio between material description words and total words). These checks could also reduce the false positives, next to the approach described in the ticket.

Identifying pages with low verb frequency could indeed help refine boreprofile detection, especially in distinguishing between actual profiles and descriptive/narrative pages containing similar terminology.