Spancat single_label model accuracy very low compared to NER

[vignesh-spericorn]

Hello,

I have trained a NER model for identifying 3 custom entities

1. Name of products which can have single or multiple words,
2. Serial number of the product
3. Quantity

The NER model performs reasonably well, but we need to also get confidence scores of prediction. So i trained a spancat single_label model (Since there are no overlapping entities) using the same training data, but the accuracy of the spancat model is nowhere near the NER model. I have attached the config file i used. Kindly suggest me on how to improve.

Annotations are done using NER annotator by tecoholic, Training was done using Google Colab

I created the config file using the command : python -m spacy init config configs/spancat_singlelabel.cfg --lang en --pipeline spancat_singlelabel --optimize efficiency
but it had a line "vectors = {"@vectors":"spacy.Vectors.v1"}" under [nlp] which threw an error during training so i deleted it and then the training ran fine.

config file : spancat_singlelabel_efficiency.txt
conversion of annotations to train.spacy : convert-annotations-to-spacy.txt

Thanks in Advance

[shadeMe]

Could you run the spacy debug data command with your converted annotation data and post the output here?

[vignesh-spericorn]

Please find the output for spacy debug data command

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, spancat_singlelabel
7126 training docs
7126 evaluation docs
⚠ 6202 training examples also in evaluation data

============================== Vocab & Vectors ==============================
ℹ 87976 total word(s) in the data (12897 unique)
ℹ No word vectors present in the package

============================ Span Categorization ============================

Spans Key Labels

---

sc {'DESCRIPTION', 'QUANTITY', 'PART NUMBER'}

ℹ Span characteristics for spans_key 'sc'
ℹ SD = Span Distinctiveness, BD = Boundary Distinctiveness

Span Type Length SD BD N

---

PART NUMBER 2.87 0.71 0.99 9942
DESCRIPTION 2.54 1.14 1.13 7073
QUANTITY 1.00 1.97 1.38 6774

---

Wgt. Average 2.24 1.20 1.14 -

ℹ Over 90% of spans have lengths of 1 -- 7 (min=1, max=15). The most
common span lengths are: 1 (47.05%), 2 (6.58%), 3 (19.75%), 5 (10.32%), 7
(8.93%). If you are using the n-gram suggester, note that omitting infrequent
n-gram lengths can greatly improve speed and memory usage.
✔ Spans are distinct from the rest of the corpus
✔ Boundary tokens are distinct from the rest of the corpus
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels

================================== Summary ==================================
✔ 6 checks passed
⚠ 1 warning

[shadeMe]

Sorry about the late reply. The training stats show that over 85% of the training data is also present in the evaluation set, so the validation scores you see during the training process will not be indicative of the model's actual performance on unseen data. You'll want to ensure that there is little to no overlap between the training, evaluation and test datasets.

For more context and information on how to interpret the model's performance and tips to improve performance, please refer to this post.

[vignesh-spericorn]

Thank you for the reply. Will check it out