Multi-Word Expression Tokenization #3
Description
Description
Enable rule-based tokenization that regroups neighboring tokenized terms that logically belong together under the same entities. Think compound words or full names.
Objectives
- Edit tokenization functions to allow a variable number of parameters to allow for specific rules and exceptions during tokenization.