Pass context in custom pipeline component

[thomasthiebaud]

Is it possible to access the context in a custom component so the custom component can be used to set an extension? Something like that

class CustomComponent:
  def __call__(self, doc):
    doc._.custom = False

  def pipe(self, docs, as_tuples=False):
    if as_tuples:
      for doc, context in docs:
        doc._.custom = context
    else:
      for doc in docs:
        doc._.custom = False
    

nlp = spacy.black("en")
nlp.add_pipe(CustomComponent())
for doc in nlp.pipe([("This is a sentence.", True)], as_tuple=True):
  print(doc._.custom)

Otherwise is there an alternive way to do it?

[thomasthiebaud]

A real use case would be to create a pipeline component to do a prediction based on the text and an optional property passed in the context (for example a category)

[thomasthiebaud]

Alternatively if the context was inside the Doc, I would be able to set an extension

def get_category(doc, context={}):
    return context.get("category", None)

Doc.set_extension("category", getter=get_category)

Another solution could be to change Language.pipe to accept a list of Doc instead of a list of text https://github.com/explosion/spaCy/blob/master/spacy/language.py#L744

[adrianeboyd]

External features are definitely something that's not easy to handle in the current pipeline configuration and there's a lot of room for improvement. I like the idea of having Language.pipe() skip the tokenization step if you provide a Doc, since that would be a minor change that could add a lot of flexibility. I think you'd need to add more validation at the same time so you don't accidentally combine docs/components that don't share the same vocab, since that can lead to unexpected errors. (This is a good idea either way, but the current setup makes it pretty hard for a typical user to do it incorrectly, so there aren't a lot of checks.)

With the current implementation, it's not as compact as nlp.pipe(), but you can call any pipeline component with a doc (or if it has .pipe() with docs), so you can set up your own pipeline sequence like this:

docs = nlp.tokenizer.pipe(texts)

# this could be its own pipeline component instead with a database or similar to look up the context
for doc, context in zip(docs, contexts):
    doc._.my_extension = context

nlp.get_pipe("my_component").pipe(docs)
nlp.get_pipe("tagger").pipe(docs)
nlp.get_pipe("parser").pipe(docs)

[thomasthiebaud]

That seems to work. Is there an equivalent for the tokenizer? eg piping a list of Doc to a Tokenizer? That sounds weird but will allow me to build this kind of nested pipeline where the first part detect the language and the second part uses different pipeline depending on the language

For example if I want to use spacy for en but spacy-stanza for da I would like to write something like that

docs_by_language = {}
docs = nlp.tokenizer.pipe(texts)

for doc in docs:
  doc._.lang = "" # Some clever language detection, maybe using https://spacy.io/universe/project/spacy-langdetect
  doc._.category = category

for doc in docs:
    docs_by_language[doc._.lang].append(doc)

for language, items in docs_by_language.items():
  nlp = get_pipeline(language) # Return Language for en, StanzaLanguage for da
  # ...

But this is maybe more a feature request for spacy-stanza StanzaLanguage extends Language so this is just another example

[adrianeboyd]

There's no way to pass a doc to a tokenizer. It doesn't really make sense to do so, since if you already have a doc with tokens, there's nothing for the tokenizer to do.

With spacy-stanza the whole stanza analysis is effectively packed into the "tokenizer" step of the spacy pipeline, so I don't think there's an easy way for you to configure language detection as part of a single pipeline and also use a spacy-stanza model. The StanzaLanguage pipeline uses the stanza tokenizer and multi-word expander instead of any of the built-in spacy tokenizers.

import stanza
from spacy_stanza import StanzaLanguage

snlp = stanza.Pipeline(lang="de")
nlp = StanzaLanguage(snlp)
print(nlp.pipe_names) # []

nlp = spacy.load("de_core_news_sm")
print(nlp.pipe_names) # ['tagger', 'parser', 'ner']

I think you'll probably want to do language detection outside of spacy just on the plain texts (I don't think that tokenization will make much of a difference for language detection and you wouldn't want to spend time tokenizing and creating spacy docs twice) and then have a method that calls the right spacy pipeline for each text.

[thomasthiebaud]

Thanks for the details. I did that and it works ok. I still think it would be nice to build end to end pipeline with Spacy, but I understand that this is maybe not really in your scope.