spanruler + spancat inconsistencies

[micmizer]

Maybe I am mis-interpreting the documentation but I am seeing some odd behavior with spanruler + spancat that I would like to understand better.

My understanding was that entity ruler + ner could improve ner's training and the same could be said for spanruler + spancat.

When i have Pipeline: ['tok2vec', 'spancat'] i get a score of .70 with my current training data set:

E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  ----------  ----------  ----------  ------
 82    4400          0.83        879.74       68.15       74.19       63.01    0.68
 87    4600          0.39        876.03       66.14       79.05       56.85    0.66
 93    4800          0.63        868.66       69.57       82.24       60.27    0.70

When i have Pipeline: ['tok2vec', 'span_ruler', 'spancat'] i get similar F/P/R scores but a score of .35 using the same training/dev data set?

E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_RULER_F  SPANS_RULER_P  SPANS_RULER_R  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  -------------  -------------  -------------  ----------  ----------  ----------  ------
203    8600          0.79        799.87           0.00           0.00           0.00       72.06       77.78       67.12    0.36
208    8800          0.38        772.77           0.00           0.00           0.00       71.70       79.83       65.07    0.36
214    9000          0.38        784.71           0.00           0.00           0.00       69.85       75.40       65.07    0.35

In experimenting with the trained model that had span_ruler before spancat in the pipeline it seems the model is removing spans that should be set by the span_ruler (almost seems like the patterns are not working at all)? If i remove span_ruler from the trained pipeline and re-add it at the end i find the patterns are working as expected.

I was hoping my patterns could improve spancats training but it seems to have the opposite effect. If i remove span_ruler all together the predictions from spancat seem better then when span_ruler(ahead of spancat) is in the pipeline?

Here is my current configuration:

[paths]
train = null
dev = null
vectors = "en_core_web_lg"
init_tok2vec = null
raw_text = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","span_ruler","spancat"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
vectors = {"@vectors":"spacy.Vectors.v1"}

[components]

[components.span_ruler]
factory = "span_ruler"
annotate_ents = false
ents_filter = {"@misc":"spacy.first_longest_spans_filter.v1"}
matcher_fuzzy_compare = {"@misc":"spacy.levenshtein_compare.v1"}
overwrite = false
phrase_matcher_attr = null
spans_filter = null
spans_key = "sc"
validate = false

[components.span_ruler.scorer]
@scorers = "spacy.overlapping_labeled_spans_scorer.v1"
spans_key = "sc"

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_range_suggester.v1"
min_size = 1
max_size = 8

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.pretrain]
@readers = "spacy.JsonlCorpus.v1"
path = ${paths.raw_text}
min_length = 5
max_length = 500
limit = 0

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = ["span_ruler"]
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0
spans_ruler_f = 1.0
spans_ruler_p = 0.0
spans_ruler_r = 0.0

[pretraining]
max_epochs = 1000
dropout = 0.2
n_save_every = null
n_save_epoch = null
component = "tok2vec"
layer = ""
corpus = "corpora.pretrain"

[pretraining.batcher]
@batchers = "spacy.batch_by_words.v1"
size = 3000
discard_oversize = false
tolerance = 0.2
get_length = null

[pretraining.objective]
@architectures = "spacy.PretrainCharacters.v1"
maxout_pieces = 3
hidden_size = 300
n_characters = 4

[pretraining.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = true
eps = 0.00000001
learn_rate = 0.001

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.span_ruler]

[initialize.components.span_ruler.patterns]
@readers = "srsly.read_jsonl.v1"
path = "../patterns/no_sentiment_patterns.jsonl"
skip = false

[initialize.components.spancat]

[initialize.components.spancat.labels]
@readers = "spacy.read_labels.v1"
path = "../train/labels/spancat.json"

[initialize.tokenizer]

[adrianeboyd]

Remove the score weights for span_ruler_* to remove them from the score calculation:

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

The score weights settings aren't adjusted automatically when you change the spans_key for span_ruler, so you have to do this by hand in your config.

[micmizer]

Hi adrianeboyd. Removing the span_ruler_* entries from the score weights did not have any impact on the score.

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

spancat before span_ruler

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'spancat', 'span_ruler']
ℹ Set annotations on update for: ['span_ruler']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SPANS_RULER_F  SPANS_RULER_P  SPANS_RULER_R  SCORE 
---  ------  ------------  ------------  ----------  ----------  ----------  -------------  -------------  -------------  ------
  0       0        119.77       1184.60        0.11        0.06       66.44           0.00           0.00           0.00    0.00
...
...
105    5200          0.95       1497.33       67.10       63.41       71.23           0.00           0.00           0.00    0.34
110    5400          1.08       1440.86       68.65       66.24       71.23           0.00           0.00           0.00    0.34
116    5600          0.83       1261.67       67.97       65.00       71.23           0.00           0.00           0.00    0.34

spancat after span_ruler

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'span_ruler', 'spancat']
ℹ Set annotations on update for: ['span_ruler']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_RULER_F  SPANS_RULER_P  SPANS_RULER_R  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  -------------  -------------  -------------  ----------  ----------  ----------  ------
  0       0        119.77       1184.60           0.00           0.00           0.00        0.06        0.03       33.56    0.00
...
...
 70    4000          0.55       1589.57           0.00           0.00           0.00       57.89       80.49       45.21    0.29
 76    4200          0.44       1554.17           0.00           0.00           0.00       58.23       75.82       47.26    0.29
 82    4400          0.30       1564.80           0.00           0.00           0.00       58.09       73.68       47.95    0.29

Only spancat (using prodigy db-out config)

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'spancat']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  ----------  ----------  ----------  ------
  0       0        123.32       1087.32        0.05        0.03       26.03    0.00
...
...
 82    4400          0.83        879.74       68.15       74.19       63.01    0.68
 87    4600          0.39        876.03       66.14       79.05       56.85    0.66
 93    4800          0.63        868.66       69.57       82.24       60.27    0.70

[adrianeboyd]

Try setting all of the spans_ruler_* scores to to null instead of removing them? There will be minor differences in the results with the added span_ruler, but in all the examples this looks like a math/scoring problem and not an actual training problem. (There is no key ruler so a 0.0 score is getting averaged in.)

I would also add that I'd have to double-check exactly how the scoring works if the span_ruler and spancat both add the same span more than once. spancat will overwrite the entire spans_key, but I think that span_ruler would extend the existing spans using the default settings.

[micmizer]

Setting spans_ruler_* to null appears to have done the trick, thanks!

ruler before

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'span_ruler', 'spancat']
ℹ Set annotations on update for: ['span_ruler']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  ----------  ----------  ----------  ------
  0       0        119.77       1184.60        0.06        0.03       33.56    0.00
...
...
 70    4000          0.55       1589.57       57.89       80.49       45.21    0.58
 76    4200          0.44       1554.17       58.23       75.82       47.26    0.58
 82    4400          0.30       1564.80       58.09       73.68       47.95    0.58

ruler after

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'spancat', 'span_ruler']
ℹ Set annotations on update for: ['span_ruler']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  ----------  ----------  ----------  ------
  0       0        119.77       1184.60        0.11        0.06       66.44    0.00
  0     200        469.28       5437.82       55.94       63.48       50.00    0.56
...
...
105    5200          0.95       1497.33       67.10       63.41       71.23    0.67
110    5400          1.08       1440.86       68.65       66.24       71.23    0.69
116    5600          0.83       1261.67       67.97       65.00       71.23    0.68

I would also add that I'd have to double-check exactly how the scoring works if the span_ruler and spancat both add the same span more than once. spancat will overwrite the entire spans_key, but I think that span_ruler would extend the existing spans using the default settings.

My observations from testing the outputted models that appears to be the case. If span_ruler is before spancat then some of the ruler pattern matches are getting removed by spancat. If span_ruler is after spancat it does not overwrite spancat but is supplemental.

I was hopping that having span_ruler ahead of spancat would improve the models training but so far that dosent appear to be the case. Is this expected?

[adrianeboyd]

It can go either way, but a high-precision rule can work better directly than trying to have the model learn the rule.

For the scorer it shouldn't matter if the same spans are repeated, since it compares sets of spans underneath. But if you need to for further processing you can use a spans_filter in the span_ruler to filter duplicates.