Voice swapping via mel spectrogram

[Herajy]

Hi,

I would like to swap voice A into voice B. I have tried Nemo fine-tuning, but it needs a bit of time for training and fine-tuning.

My idea is that to generate mel spectrogram for Voice A (say mel 1) and then generate mel spectrogram for Voice B (say mel2). So we will have something like this:

audio1 -> mel1 -> mel2 -> audio2

My question (s):
(1) will this be reasonable
(2) if so, is there some tool in Nemo to do such conversion between mel1 to mel2

Thank you in advance for any hint about this

[gedefet]

There is no need for doing that.

You can select, after training, which voice you want to synthesize:

def str_to_audio(inp, pace=1.0, durs=None, pitch=None):
    y = torch.randperm(1)
    y[0] = 1
    y = y.to(torch.int)
    with torch.no_grad():
        tokens = fastpitch.parse(inp)
        spec, _, durs_pred, _, pitch_pred, *_ = fastpitch(text=tokens, durs=durs, pitch=pitch, speaker=y, pace=pace)
        audio = hifigan.convert_spectrogram_to_audio(spec=spec).to('cpu').numpy()
    return spec, audio, durs_pred, pitch_pred

where y is the speaker

[gedefet]

Of course, you have to train with all the speakers.

[Herajy]

Thank you @gedefet for your answer. But according to your suggested idea I have to train/fine-tune for all speakers, correct? But based on my experimentation fine-tuning/training require some times and also need to collect 5-6 hrs for each speaker to get an acceptable output.

[gedefet]

Look at the end of https://colab.research.google.com/github/NVIDIA/NeMo/blob/stable/tutorials/tts/FastPitch_Finetuning.ipynb:

{"audio_filepath": "new_speaker.wav", "text": "sample", "duration": 2.6, "speaker": 1} {"audio_filepath": "old_speaker.wav", "text": "LJSpeech sample", "duration": 2.6, "speaker": 0} 5 hours of data from the old speaker should be sufficient. Since we should have less data from the new speaker, we need to ensure that the model sees a similar amount of new data and old data. For each sample from the old speaker, please add a sample from the new speaker in the .json. The samples from the new speaker will be repeated.

Modify the fastpitch training command to point to the new training and validation .jsons, and update model.n_speakers=1 to model.n_speakers=2. Ensure the pitch statistics correspond to the new speaker.

For HiFiGAN finetuning, the training should be done on the new speaker data.

And from my own experience, you have to find the right balance between the amount of data of each dataset.

[Herajy]

ok, I will try this. Many thanks for your reply