🚀 The feature

If we want to train / provide multilingual recognition models we need to extend our predefined vocabularies

reference PR: https://github.com/mindee/doctr/pull/1355/files

Here’s your updated checklist, extended with the missing languages from the list you provided (excluding the ones you've already marked as done):

Absolutely! Here's your list split by writing script. Some languages use multiple scripts (like Serbian or Uzbek), and I’ve grouped them accordingly.

Cyrillic

• tajik [datasets] Add several cyrillic based vocabs #1925
• belarusian [datasets] Add several cyrillic based vocabs #1925
• bulgarian [datasets] Add several cyrillic based vocabs #1925
• belarusian [datasets] Add several cyrillic based vocabs #1925
• kazakh [datasets] Add several cyrillic based vocabs #1925
• kyrgyz [datasets] Add several cyrillic based vocabs #1925
• macedonian [datasets] Add several cyrillic based vocabs #1925
• mongolian [datasets] Add several cyrillic based vocabs #1925
• serbian (cyrillic) [datasets] Add several cyrillic based vocabs #1925
• uzbek (cyrillic) [datasets] Add several cyrillic based vocabs #1925

Latin

• serbian (latin) [datasets] Extend latin based vocabs #1926
• afrikaans [datasets] Extend latin based vocabs #1926
• azerbaijani [datasets] Extend latin based vocabs #1926
• bosnian [datasets] Extend latin based vocabs #1926
• catalan [datasets] Extend latin based vocabs #1926
• welsh [datasets] Extend latin based vocabs #1926
• estonian [datasets] Extend latin based vocabs #1926
• basque [datasets] Extend latin based vocabs #1926
• frisian [datasets] Extend latin based vocabs #1926
• irish [datasets] Extend latin based vocabs #1926
• scottish gaelic [datasets] Extend latin based vocabs #1926
• galician [datasets] Extend latin based vocabs #1926
• hausa [datasets] Extend latin based vocabs #1926
• indonesian [datasets] Extend latin based vocabs #1926
• icelandic [datasets] Extend latin based vocabs #1926
• lithuanian [datasets] Extend latin based vocabs #1926
• latvian [datasets] Extend latin based vocabs #1926
• malagasy [datasets] Extend latin based vocabs #1926
• malay [datasets] Extend latin based vocabs #1926
• romanian [datasets] Extend latin based vocabs #1926
• slovak [datasets] Extend latin based vocabs #1926
• slovenian [datasets] Extend latin based vocabs #1926
• somali [datasets] Extend latin based vocabs #1926
• albanian [datasets] Extend latin based vocabs #1926
• swahili [datasets] Extend latin based vocabs #1926
• tagalog [datasets] Extend latin based vocabs #1926
• uzbek (latin) [datasets] Extend latin based vocabs #1926
• zulu [datasets] Extend latin based vocabs #1926

Arabic Script

• persian
• pashto
• urdu
• sindhi
• uyghur
• kurdish (arabic)

Devanagari

• marathi
• nepali
• sanskrit

Other Brahmic Scripts

• assamese
• kannada
• malayalam
• oriya
• punjabi
• tamil
• telugu
• sinhala

Javanese Script

• javanese

Sudanese Script

• sundanese

Georgian Script

• georgian

Armenian Script

• armenian

Hebrew Script

• yiddish

Ethiopic Script

• amharic
• oromo

Thaana / Burmese / Other Unique Scripts

• burmese
• lao
• thai
• khmer

East Asian Scripts

• japanese
• chinese (simplified)
• korean (hangul)

Let me know if you'd like a markdown file version of this, or if you want visual icons, group headers, or collapsible sections!

Let me know if you'd like this grouped, sorted alphabetically, or saved to a file!

For example https://github.com/eymenefealtun/all-words-in-all-languages could be used to extract language specific charsets

The current multilingual vocabs entry can be extended with the new created language entries to provide a deduplicated list of a most complete multilingual char representation

latin_extended (german, spanish, czech, and so on), cyrillic and hebrew should be really low hanging fruits to include for training

ressources:

https://sites.google.com/site/worldfactsinc/Non-Latin-Script-Languages-Of-The-World

https://www.omniglot.com/writing/langalph.htm