Training for text detection isn't giving expected results

[akashlp27]

Hey, I have prepared the training data mentioned in the Readme in the References/Detection using the ICDAR scene text detection data, I have trained the model with 5000 images and around 30 epoch with db_resnet50 architecture, but the results isn't improving and the output isn't upto the mark, any suggestions to improve or finetune would help a lot, thanks. I'm training with pytorch version

[Atomme1]

Have you tried using the --pretrained True feature ?

[akashlp27]

Yes, I'm loading the pretrained weight file and resuming from it

[akashlp27]

from doctr.models import ocr_predictor
from doctr.io import DocumentFile
import torch
import tensorflow as tf
import cv2
import matplotlib.pyplot as plt
import os
import numpy as np
os.environ['USE_TORCH'] = '1'

det_arch = 'db_resnet50'
predictor = detection_predictor('db_resnet50', pretrained=not isinstance(r'C:\Users\user\PycharmProjects\Doctr_train\doctr\references\detection\db_resnet50_20220531-154702.pt', str), assume_straight_pages=True, preserve_aspect_ratio=True)

single_img_doc = DocumentFile.from_images(r"C:\Users\user\Downloads\kannada\Kannada-University-Application-Form-2_crp.jpg")`

processed_batches = predictor.det_predictor.pre_processor([single_img_doc[0]])
out = predictor.det_predictor.model(processed_batches[0], return_model_output=True)
boxes_pred = out['preds']
boxes_pred = np.concatenate((boxes_pred[0].min(axis=1), boxes_pred[0].max(axis=1)), axis=-1)
seg_map = out["out_map"]
seg_map = torch.squeeze(seg_map[0, ...], 0)

seg_map = cv2.resize(seg_map.detach().numpy(), (single_img_doc[0].shape[1], single_img_doc[0].shape[0]),
                     interpolation=cv2.INTER_LINEAR)
fig, ax = plt.subplots()
ax.imshow(seg_map)
ax.axis('off')
plt.show()

This is the prediction code i'm using, any inputs would help a lot , attached is the output from the trained model

[Atomme1]

det_arch = 'db_resnet50'
predictor = detection_predictor('db_resnet50', pretrained=not isinstance(r'C:\Users\user\PycharmProjects\Doctr_train\doctr\references\detection\db_resnet50_20220531-154702.pt', str), assume_straight_pages=True, preserve_aspect_ratio=True)

Are you sure the model is well charged?
Because i had to go on the user/.cache/doctr/models and replace the file in order to use the training i did for the recognition.
Maybe try with the tensor training and see the difference ?

Hope you find your answers ❤️

[felixdittrich92]

Hi @akashlp27,
sry for the late reply :)

1. About your data i would recommend to use a bigger dataset for example a random split from SynthText (~200K) and validate on ICDAR dataset (this small datasets are mostly used for the validation part and not to train AND the pretrained models was trained only on regular text scenes (@frgfm @charlesmindee correct me if i'm wrong) but you want to detect also irregular text (5K images sounds like ICDAR2019 !?? which contains irregular text))
2. @Atomme1 got it your model is never loaded:

Example for current pip version (0.5.1)
#568 (take a look at the answer :) )

PyTorch Example with next release (0.6.0 currently on main)

model = db_resnet50(pretrained=False)
model.load_state_dict('path-to-your-model.pt', map_location='cpu')
# only detection
detector = detection_predictor(model, pretrained=False)
# full pipe
predictor = ocr_predictor(det_arch=model, pretrained=True)

[akashlp27]

Hey @frgfm ,Understood thanks, currently running my 25th epoch, yes now I can see the difference, the training loss isn't varying a much from iteration to iteration. will wait for more epochs and then conclude it, thanks.

[akashlp27]

@frgfm Hey, i have an update, the validation loss is keeping on increasing attached is the metrics,
Epoch 28/50 - Validation loss: 2.67893 (Recall: 53.16% | Precision: 66.53% | Mean IoU: 57.55%) Epoch 29/50 - Validation loss: 2.64776 (Recall: 57.03% | Precision: 63.27% | Mean IoU: 54.79%) Epoch 30/50 - Validation loss: 2.66453 (Recall: 55.53% | Precision: 63.42% | Mean IoU: 55.14%) Epoch 31/50 - Validation loss: 2.76906 (Recall: 54.14% | Precision: 65.62% | Mean IoU: 56.84%) Epoch 32/50 - Validation loss: 2.74085 (Recall: 56.59% | Precision: 65.04% | Mean IoU: 56.77%) Epoch 33/50 - Validation loss: 2.89126 (Recall: 54.94% | Precision: 67.65% | Mean IoU: 58.58%) Epoch 34/50 - Validation loss: 2.91126 (Recall: 55.88% | Precision: 64.66% | Mean IoU: 56.14%) Epoch 35/50 - Validation loss: 2.9188 (Recall: 55.38% | Precision: 65.46% | Mean IoU: 56.95%) Epoch 36/50 - Validation loss: 3.0328 (Recall: 55.92% | Precision: 63.21% | Mean IoU: 55.08%) Epoch 37/50 - Validation loss: 3.13755 (Recall: 55.41% | Precision: 66.16% | Mean IoU: 57.30%) Epoch 38/50 - Validation loss: 3.12072 (Recall: 55.63% | Precision: 65.37% | Mean IoU: 57.03%) Epoch 39/50 - Validation loss: 3.18182 (Recall: 56.37% | Precision: 65.43% | Mean IoU: 56.90%) Epoch 40/50 - Validation loss: 3.18879 (Recall: 56.12% | Precision: 64.67% | Mean IoU: 56.23%) Epoch 41/50 - Validation loss: 3.26834 (Recall: 55.21% | Precision: 64.77% | Mean IoU: 56.52%)
the reason which the checkpoint isn't saving because of the validation loss

[felixdittrich92]

Hi @akashlp27 some question to get more insights :)

• Can you share the command you have used to run the training script (to get information about lr, weight decay, ...) or share logs from W&B (<-- would be really helpful)
• You have used the SynthText dataset ? have you used the integration from docTR ? How many train/val samples ?
• If i understood it correctly the loss increases after n epochs correctly ?

Some things i would try: don't freeze the backbone, if pretrained start with a lower lr like 1e-4, set a weight decay from 0.1,

[akashlp27]

@felixdittrich92 , Thanks for asking,

1. I am using some part of SynthText like around 15k samples for training and and 2.5k samples for validation, No it isn't from docTR's integration. Sorry, I don't have any wandb logs, as I set it to False in the params.
2. I am using the default Learning rate : 1e-3 and weight decay 0, and pretrained = True. batch size =4, image_size=1024
3. At the very beginning, the training loss was varying a lot after each iteration, after 25 epochs, it is being decreasing , but the validation loss is increasing right from the 1st epoch.

yess, will try to experiment with the lr and the weight decay. And I'm using less data since I'm loading the pretrained weights file, is it a concern , not using a larger data?

[felixdittrich92]

@akashlp27 In the case of SynthText or any other dataset which aims to provide "scene text in the wild" yes ! @frgfm @charlesmindee correct me if i'm wrong:
the pretrained model was only trained on text documents which mostly do not have any major shifts or curvatures in the text.

I think 200K train and 40K val could be a good starting point