Guidance on testing model performance when training custom NER entities on a base model

[rahulraina7]

I have been able to re-train the base en_core_web_lg model with my custom entities for NER. However, I would like to evaluate its NER performance so that I avoid the [catastrophic-forgetting(https://explosion.ai/blog/pseudo-rehearsal-catastrophic-forgetting) problem.

I have tested it on conll2003 dataset and the NER results look fine. However, I believe this might be different for different test sets. Are there any guidelines to test your custom trained model compared to original pre-trained models.

Info about spaCy

• spaCy version: 2.2.3
• Platform: Darwin-18.7.0-x86_64-i386-64bit
• Python version: 3.7.6

[svlandeg]

Hi @rahulraina7, except for testing on existing datasets like you're already doing, you could also grab a bunch of unannotated text relevant to your specific domain / use-case, and run the "normal" en_core_web_lg model on it. Now use those predictions as "silver" annotations. If you'd evaluate en_core_web_lg on it, you'd get an artificial 100% accuracy. Output the performance by entity type.

Then, when you start retraining, you can rerun this evaluation with your retrained model, and see how the performance for the different entity categories changes. If some drop in performance, you know you need to insert more instances of that type during retraining.