How to train multispeaker Fastpitch?

[dtemnov]

Hi,
A preptrained multispeaker Fastpitch has been there for a while and there is also a useful hint in Fastpitch_Finetuning tutorial about mixing data of 2 speakers to enhance the quality but there is still no clear guide on training a custom multispeaker model. I'm curious if anyone tried to reproduce the results of Nemo team for more than 2 speakers(basically similar to provided multiskear Fastpich)?

[subhankar-ghosh]

@dtemnov Thank you for your question.
Training multi-speaker FastPitch is similar to training single-speaker FastPitch. Following are the changes you need to make:

1. Manifest: Your manifest file should have speaker information. For each datapoint/json object, include a field speaker: <speakerID> for example the datapoints in your manifest file should look like:

{"audio_filepath": "<path/to/audio/file1>", "duration": <duration1>, "text": "This is a valid example datapoint1", "speaker": 1}
{"audio_filepath": "<path/to/audio/file2>", "duration": <duration2>, "text": "This is a valid example datapoint2", "speaker": 1}
{"audio_filepath": "<path/to/audio/file3>", "duration": <duration3>, "text": "This is a valid example datapoint3", "speaker": 0}
.
.
.

Notice the speaker field, I have used 0 and 1 for example but they can be any integers.

2. In your training command or in yaml config file add speaker_id to the sup_data_types. Example sup_data_types: [ "align_prior_matrix", "pitch","speaker_id" ]

3. In your training command add model.n_speakers=<maximum value of the speaker IDs + 1> or in yaml config file change n_speakers: <maximum value of the speaker IDs> + 1. For example, in the above 2 speaker training the maximum value of speaker is 1 so I would set n_speaker: 2. This is because in multi-speaker training, FastPitch initializes a torch.nn.Embedding layer with n_speaker number of embeddings, see https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/modules/fastpitch.py#L166
   and speakerID are used as index for these embeddings during training and inference, see https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/modules/fastpitch.py#L250

I hope this answers your question.

[dtemnov]

I have found this example for Spanish - https://github.com/NVIDIA/NeMo/blob/main/examples/tts/conf/es/fastpitch_align_44100_ipa_multi.yaml
Looks like I will need to create a separate file with statistics for each speaker, right? (https://github.com/NVIDIA/NeMo/blob/main/scripts/tts_dataset_files/openslr_es/pitch_stats.json)

[rlangman]

Before training you will need to run extract_sup_data.py. There is an example of how to do this in our German tutorial.

https://github.com/NVIDIA/NeMo/blob/main/scripts/dataset_processing/tts/extract_sup_data.py
https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_GermanTTS_Training.ipynb

This script will print out the global pitch mean and standard deviation across all speakers. For both single and multi-speaker training you can provide these values in the config:

https://github.com/NVIDIA/NeMo/blob/main/examples/tts/conf/fastpitch_align_44100.yaml#L18-L19

For multi-speaker training you can also provide a statistics file with the stats for each speaker (as you saw in the Spanish config). After running extract_sup_data.py, run compute_speaker_stats.py to create the necessary file (the "default" field contains the global stats).

https://github.com/NVIDIA/NeMo/blob/main/scripts/dataset_processing/tts/compute_speaker_stats.py#L21-L24

Using the statistics for each speaker is a recently added feature that we are still testing the accuracy of.

[dtemnov]

Thanks for your valuable responses! The last question, I guess, is it ok to finetune the existing fastpitch_multispeaker model with new custom voices or should I run the trainng from scratch using all HiFiTTS data? In case if it is possible and i won't lose existing HiFiTTS embeddings, how should I pass the data of a new speakers into model?

[subhankar-ghosh]

@dtemnov You can finetune the existing multispeaker model with the custom voices - this will not alter the existing HiFiTTS speaker embeddings but the weights of the model (encoder, decoder, predictors) will be changed.So, this might affect the existing speaker voice quality a bit.

I would not recommend training with HiFiTTS from scratch, as that would take a lot of time and resources.

Instead, what I would recommend is taking your custom data and mix it with a small sample from HiFiTTS dataset. Using this mixed dataset finetune the multi-speaker model. This approach would not let the model totally forget the existing speakers.

[dtemnov]

Ok, sounds good. Thanks for your help, I'll try everything and come back with more questions :)