Extend to user-provided data

[jstammers]

Hi @NickCrews, after encountering similar frustrations with libpostal and using spacy for other NLP tasks, I'm very interested by this repo.

I have a lot of proprietary address data from different countries that I would like to use to train NER models using this repo as a framework.

Are there any plans to extend this to facilitate the training of address parsers using non-US and/or non-english language addresses?

For testing, I think I should be able to do this by creating the appropriate datasets in train.nerf and dev.nerf, but I expect there are some other parts that may need modifying.

Happy to help contribute to this if needed!

[NickCrews]

Hey @jstammers, sorry this fell through the cracks and I'm not responding until now.

I don't really want to become responsible for maintaining the extra data. I am fine maintaining the actual code and scripts on a best-effort basis, as I'm doing here. I would be willing to make minor adjustments to the code/scripts here so that you/3rd parties can pull in their own data and config.cfg but use these scripts. Would that work? I think you can do this currently, just by modifying the existing justfile recipe as a separate bash script:

python nerf.py yourdata/train.nerf yourdata/train.spacy
python nerf.py yourdata/dev.nerf yourdata/dev.spacy
mkdir -p models/yourmodel/training
python -m spacy train models/yourmodel/config.cfg --output models/yourmodel/training --paths.train ./yourdata/train.spacy --paths.dev ./yourdata/dev.spacy  --gpu-id {{gpu}}

If there are changes that would make this easier, I'm down to hear.

Honestly, if I were you, I might just copy this whole repo, start from there, and never worry about pulling in new changes. It might just be easier for you if you had a totally separate branch that you have complete control over.