Allowing NER to overwrite existing entities

[NixBiks]

1. I have a pipeline with chained pattern matching to mark spans as certain entities. On top of that I want to add a trained NER model that should be allowed to overwrite existing entities. It isn't clear to me if that is possible atm? I realize that I could add NER in front and then have my chained rulers not overwrite but the point of having them chained is to override - I could hack my way out of it though.

2. One more thing that isn't clear to me; I've trained my NER model using prodigy and and saved the output to disk. How do I load just the EntityRecognizer from disk. I've tried

ner = EntityRecognizer(nlp.vocab)
ner.from_disk('/path/model')

and

ner = EntityRecognizer(nlp.vocab)
ner.from_disk('/path/model/ner')

but both fails. Either FileNotFoundError: [Errno 2] No such file or directory: '/path/model/moves' or ValueError: Can't read file: /path/model/ner/strings.json.

Which page or section is this issue related to?

EntityRecognizer, Saving and Loading and EntityRecognizer API

[adrianeboyd]

1. No, the NER model doesn't have a setting to overwrite entities, but it will preserve any existing entities, so you could reverse your pipeline to run the higher priority models first. The only complication is that the presence of existing entities can change the predictions to some degree (and especially if they're otherwise overlapping), so you want to check that the performance is what you expect. In particular, you can end up with partial entities because it predicts word by word. It can start predicting an entity a few words before it runs into an existing entity, and then it leaves the partial entity when it moves on to the next word. So it can predict something like B-PER I-PER B-OTHER I-OTHER L-OTHER when OTHER is already there, which might not be what you want.

   So if the logic is more complicated or the behavior with overlapping entities is causing problems, you may want to run each separately and then merge to one set of non-overlapping entities with your own custom method / priorities. The entity rulers do remove existing overlapping entities rather than predicting partial entities, but the current NER model can't really do this, so the only option is to predict separately (save doc.ents, reset it for the doc, and predict) and merge. It wouldn't be too difficult to write a custom pipeline component that does this so it's integrated into your pipeline.

2. You need to exclude the vocab since it's already been loaded separately from a different location in the saved model:

ner.from_disk('/path/model/ner', exclude=["vocab"])