How much CPU RAM need to training ~20K hr audio data

[mehadi92]

Hi, I'm trying to fine-tune the en_conformer_ctc_large model using ~20K hr audio data. I'm using 4 a100 40 GB GPU with 335 GB RAM but after ~1600 steps I'm getting an error

ERROR: unexpected bus error encountered in workers. this might be caused by insufficient shared memory (shm)

The above error is coming because the 335 GB RAM gets full, Here is my trainer,

trainer = ptl.Trainer(
    accelerator=accelerator,
    max_epochs=EPOCHS,
    accumulate_grad_batches=2,
    log_every_n_steps=10,
    check_val_every_n_epoch=1,
    strategy="ddp_find_unused_parameters_false",
    logger=False,
    enable_checkpointing=False,
    devices=-1,
    precision=16,
)

I have used data bucketing (4 buckets, 1024 shards) for train data. also use pin_memory=True, the max audio length was 18.5 seconds

[mehadi92]

@titu1994 any idea regarding this - Thanks

[OllieBroadhurst]

Are you using Docker? See if this helps.

[mehadi92]

@OllieBroadhurst Yes i'm using docker. And I already set -ipc=host