Is it possible to loading ASR data faster for ~20k hr data and make training faster?

[mehadi92]

I try to train the ASR model with 2TB of data. It takes around 50 min to start actual training. Could we make it faster? I'm using 16 a100 40 GB GPU.

Do you have any suggestions for training the ASR model with 2TB data to efficiently use GPU and faster training?

I'm using the conformer transducer medium model and it was taking ~6hr for 1 epoch without data bucketing.
I'm already using mixed precession(bf16)

Thanks

[titu1994]

Startup time is for manifest processing. You can try using sharded manifests, but it is under testing and not fully documented yet but you can use the following docs - https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/datasets.html#tarred-datasets

Ctrf-F "sharded manifest"

[VahidooX]

I also suggest to use FastConformer models instead of regular Conformer. It gives you more than 2x speedup in training and inference.

[mehadi92]

@VahidooX YES fast conformer is 2X faster

[mehadi92]

But there is no pre-train of a fast conformer model with 128 tokenizers. All fast-conformer model comes with 1024 tokens

[VahidooX]

What is the problem with vocab size 1024?

[mehadi92]

@VahidooX Not much problem with vocab size. Need to create a new tokenize with vocab size 1024. Could not use the same vocab that I can use in conformer ctc models.