Add arabic language to doctr

[sridi-Rania]

Hello , i'am working in my final year project and i want to fine tune doctr and add the arabic luangage to it but it doesen't work predection are always empty and i think the problem is due to the ctc decoder or I don't know can any one help me to add the arabic to it ?

[felixdittrich92]

Hi @sridi-Rania 👋🏼,
Could you provide a few more information what you have tried ? :)

[sridi-Rania]

In fact, Arabic letters have multiple forms depending on their position in a word. However, your Arabic vocabulary only includes the basic forms of the letters, making it incomplete. i can take an exemple these are ﺀ ﺁ ﺃ ﺅ ﺇ ﺉ all forms for the same alphabet

[felixdittrich92]

At the end it depends how utf-8 encodes the letters for example for gujarati we have had to use some "variables" because some letters was a combination of the basic letter and the circle dotted "extension" -> so one letter on paper but read by the system two.

"gujarati_vowels": "અઆઇઈઉઊઋએઐઓ",
"gujarati_consonants": "ખગઘચછજઝઞટઠડઢણતથદધનપફબભમયરલવશસહળક્ષ",
"gujarati_digits": "૦૧૨૩૪૫૬૭૮૯",
"gujarati_punctuation": "૰ઽ◌ંઃ॥ૐ઼ઁ" + "૱",

The easiest would be to read all your labels in a list, extract all unique chars and deduplicate then we could update our internal vocab wdyt ? :)

[sridi-Rania]

Yes, I've done that and increased the dataset size to 20,000, but I think the model still isn't learning properly. The issue might be due to the greedy CTC decoder used in the model ?

[felixdittrich92]

But loss looks much better now 👍

How does it look if you add --pretrained to your training command (for fine tuning) ? Because actual you train from scratch where 20k is of course to less data

[sridi-Rania]

I'll give it a try and get back to you with either success or failure. Thank you!

[sridi-Rania]

Hello Is it possible to freeze all layers except one for fine-tuning, since my dataset is small?

[sridi-Rania]

Hello, I’d like to ask something. I’m currently training the recognition model parseq using a dataset where each image contains a single word. Should I continue with this approach, or would it be better to train the model using images that contain full phrases? thank you for your help

[felixdittrich92]

Only 1 word per image we don't support lines or phrases so everything correct 👍
Btw. you mentioned that our arabic vocab isn't complete what's missing ? :)

[sridi-Rania]

Yes, in fact, in our Arabic language, each letter can take on multiple forms depending on its position within a word. I can provide you with the complete set of letter forms if needed. I'm not sure if it's the correct approach, but when I tried to train the model using the vocabulary you provided, it didn’t work. So I had to include the Arabic letters in the vocabulary file separated by spaces, in order to represent all their contextual forms and this is the only way that make the train work

[felixdittrich92]

Yeah this would be great 👍

At the end we should cover each unique "char" in our vocab so that we later on can generate each combination vowel + diacritic combination for example

[sridi-Rania]

This file contains all contextual forms of the Arabic alphabet depending on their position in words. You can check it — it might include some unnecessary forms or something like that.
alph.txt

[sridi-Rania]

I trained the model, but despite the values and precision, it failed to detect anything when I used photos from the internet. I honestly don't know what to do. Could the issue be related to the fonts? Should I try using different fonts instead of the default ones? Do you have any suggestions

[sridi-Rania]

Hello again, while trying to fine-tune the detection model with 200 examples, it didn't work. Should the images have a specific size or format, or could there be another issue? , thank you

[felixdittrich92]

Hi :)

Could you share the training command you used and if possible an example with displaying the annotations ?

under the hood everything is resized to 1024x1024 by keeping the aspect ratio and symmetric padding

[felixdittrich92]

you can start the training with --show-samples and share the image

[sridi-Rania]

HEllo again , sorry i resolved the problem but i have just a question when training the recognition model with the arabic data , I used the 'multilingual' option as an argument and a pretrained model. However, after training, when testing the model, it does not recognize the other languages I had specified. Could you please correct this issue?

[felixdittrich92]

Hey :)
It can only recognize the chars which are in your training dataset everything else will be "forgetten" because the head layer is completly reset

[felixdittrich92]

Reference datasets can be found here for both recognition and detection :)
#1654

[sridi-Rania]

hope this message finds you well. I am pleased to share that I have successfully fine-tuned the Doctr model for the Arabic language, and it is now performing well in this context.
I would be honored to contribute my model to your GitHub repository and become an active contributor to the project. I am also willing to support other users who are interested in working with Arabic, and I look forward to helping expand Doctr's multilingual capabilities.
Thank you for considering my request. I look forward to your response.

[felixdittrich92]

Hi @sridi-Rania 👋

Great to hear it’s working well now ! 👍
We’re always happy to welcome external contributors and those sharing their models - thank you! 🤗

The easiest way to share your model(s) is through our Hugging Face integration:
👉 https://mindee.github.io/doctr/using_doctr/sharing_models.html

For example:

from doctr.models import recognition, login_to_hub, push_to_hf_hub

login_to_hub()

my_vocab = "abc"  # use the same vocab your model was trained on
my_awesome_model = recognition.crnn_vgg16_bn(pretrained=False, pretrained_backbone=False, vocab=my_vocab)

# Load your checkpoint
reco_params = torch.load('<path_to_pt>', map_location="cpu")
my_awesome_model.load_state_dict(reco_params)

# Push to the Hub
push_to_hf_hub(
    my_awesome_model,
    model_name='doctr-crnn-vgg16-bn-arabic-v1',
    task='recognition',
    arch='crnn_vgg16_bn'
)

If you’re interested in contributing to docTR, feel free to join our Slack community for easier communication - I’ll ping you once you’ve joined! 😊

Best,
Felix

[sridi-Rania]

I would like to contribute to the Doctr project and join the Mindee Community workspace.
Could you please send me an invitation to join the workspace? You can use this email address for the invitation: raniasridi25@gmail.com, as I am unable to join on my own — the platform requires a @mindee.com email address.
Thank you very much for your time and consideration

[felixdittrich92]

Done :)