could Matcher using python multiprocessing ?

[meshiguge]

[honnibal]

Probably, yes. We're interested in trying multi-processing in Language.pipe() as well.

[meshiguge]

import spacy 
nlp = spacy.load('en' )
def task_builder(k,file):
      for line in file: 
          lineDoc = nlp(line) 
          matcher = Matcher( nlp.vocab ) 
          matcher.add('Pattern', None, pattern ) 
          matches = matcher( lineDoc ) 
          for each_match in matches:
               do_something 

pool = multiprocessing.Pool(process_num)
for k, file in enumerate(files):
     pool.apply_async(task_builder, args=(k, file ) )
 pool.close()
 pool.join()

@honnibal , but the code will hang out , could not work

[svlandeg]

As a quick update, language.pipe() now supports multiprocessing by setting n_process to an integer > 1.

PRs to have similar support for the Matcher are welcome!