This repository was archived by the owner on Oct 31, 2023. It is now read-only.
This repository was archived by the owner on Oct 31, 2023. It is now read-only.
Tokenization issue in to-En bilingual dictionaries #182
Description
Hi all -- fyi, there appears to be a tokenization issue in the *-to-En bilingual dictionaries. We commonly see word, -- where the comma wasn't tokenized away. I see this in de-en, fi-en, it-en, and ru-en, at least.