Anybody successfully trained NeMo VITS / LJSpeech ?

[lumpidu]

Hi,

has anybody successfully trained an LJSpeech based NeMo TTS VITS model ? I am currently training on 1 GPU because with multiple GPU's I experienced an error after ~97.000 steps, which I have opened an issue for.

The Tensorboard curves don't show yet an overall convergence with currently 75.000 steps. The paper mentions that they trained up to 800.000 steps

Here is my Tensorboard with values loss_gen_all, loss_dur:

The value loss_gen_all determines the overall quality.

Values loss_fm, loss_mel:

loss_fm probably is the reason for the sideway development of loss_gen_all as it's continously going up instead of down, whereas all other loss values converge into the right direction. The question is, how loss_fm will develop over time.

I was slightly deviating from the standard NeMo model settings and instead used the parameters as stated in the paper, as well as set the parameters pin_memory:true and precision: 16 for non-automatic mixed precision:

index 1002bdfe8..33c6be90c 100644
--- a/examples/tts/conf/vits.yaml
+++ b/examples/tts/conf/vits.yaml
@@ -98,10 +98,10 @@ model:
     
     dataloader_params:
       num_workers: 8
-      pin_memory: false
+      pin_memory: true
 
     batch_sampler:
-      batch_size: 32
+      batch_size: 64
       boundaries: [32,300,400,500,600,700,800,900,1000]
       num_replicas: ${trainer.devices}
       shuffle: true
@@ -132,7 +132,7 @@ model:
       shuffle: false
       batch_size: 16
       num_workers: 4
-      pin_memory: false
+      pin_memory: true
 
   preprocessor:
     _target_: nemo.collections.asr.parts.preprocessing.features.FilterbankFeatures
@@ -180,7 +180,7 @@ model:
   optim:
     _target_: torch.optim.AdamW
     lr: 2e-4
-    betas: [0.9, 0.99]
+    betas: [0.8, 0.99]
     eps: 1e-9
   
     sched:
@@ -189,10 +189,10 @@ model:
 
 trainer:
   num_nodes: 1
-  devices: 2
+  devices: 1
   accelerator: gpu
   strategy: ddp
-  precision: 32
+  precision: 16 # for non-automatic mixed precision

Interestingly, if changing the trainer batch size to 128, the values for loss_fm go out of the roof, so I stopped training after a few thousand steps:

Could somebody share hers/his Tensorboard and experience for comparison ?

Environment:

NeMo:    v1.6.2-1035-g440528abd (440528abd57e4728070e634c6916d364680271f8)
Python:  3.8.10
PyTorch: 1.13.1
Cuda:    12.0
Cuda driver:  525.60.13

[treacker]

Could somebody share hers/his Tensorboard and experience for comparison ?

So, I think ur graphs are actually looking fine, I had very similar trends. My batch size was 8x32

[lumpidu]

Hmm, how does this compare to your loss_gen_all value ?

[treacker]

loss_gen_all is platoing at around 30 for the whole training process

[lumpidu]

Right, same here, so how do you determine the training is good enough ? Checkpoints are not done anymore, because the loss_gen_all value doesn't go down, but jumps around 30. My last checkpoint therefore is around 100.000 steps ago. Which metrics should I watch out for ?

[treacker]

I think since we have a pretty balanced adversarial training, the only metric is listening to generated audios once in a while. So, you can see the gradual improvement in audio quality

[oytunturk]

Has anyone been able to replicate the quality achieved in the VITS paper on LJSpeech? I tried the pre-trained VITS models on NGC (regular, 44KHz) but they don't sound as good. How many steps were those models trained for? Is the bad quality due to lack of training or some issues in Nemo's VITS implementation? Also, the 44KHz model is documented as a 41000Hz model which is weird. Thanks!