TTS - Mixing datasets for FastPitch + HiFiGAN #3678
-
|
Hi All. At the end of the FastPitch_Finetuning.ipynb tutorial: https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb it suggest two different ways to improve performance, one of which is: Mix new speaker data with old speaker data but it isn't 100% clear how to do that. I found another FastPitch_Finetuning.ipynb here: Which is different than the other. Is this true? Or do I need something more that is not there?Why it is abandoned? Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 9 replies
-
|
@Oktai15 any input ? |
Beta Was this translation helpful? Give feedback.
-
|
@redoctopus can you please comment on this? |
Beta Was this translation helpful? Give feedback.
-
|
I've updated the FastPitch fine-tuning notebook in #3954, hopefully the procedure for mixing datasets should be more clear. As mentioned in the tutorial, we don't explicitly have a code section where you can run this mixing in the notebook itself due to resource limitations on Colab. But if you have a local machine to use, you should be able generate a new manifest and re-run the fine-tuning commands with the few command line changes specified without too much trouble (I hope!). |
Beta Was this translation helpful? Give feedback.
-
|
Hello! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, please see the finetuning tutorial mentioned in the original question, specifically the last section "Adding more data": https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb
We usually do a 1:1 ratio of old to new entries in the manifest, repeating the new entries as necessary to keep the dataset balanced, until you have the total number of samples that you want to train with. (See the tutorial for an example.) I don't think we've done any formal experiments on the effect of data mixing and scaling the dataset, but the mixing will definitely benefit smaller new speaker datasets more dramatically than for larger datasets. 15 minutes to about an hour of new speaker audio, the mixing helps a lot; after that it becomes less noticeable. And of course it also depends on whether you care about keeping quality of the original speaker's generated speech. |
Beta Was this translation helpful? Give feedback.
-
|
@redoctopus , why do you match the number of samples instead of the total duration of the datasets? I had better results matching duration instead of # of samples. I have even better results decreasing the total old data wrt the new speaker, even with little new speaker data (~25min) |
Beta Was this translation helpful? Give feedback.
-
|
Duration is certainly more precise! Interleaving them for an equal number of samples was just simpler to write.
This is interesting, thanks for bringing it up. We haven't done much in the way of experimentation for this--mostly we were comparing "no mixing (new speaker data only)" vs. "1:1 mixing (new and old speaker data)" and saw that the mixing was noticeably better for most of our new speakers (i.e. 15m, 30m, 60m of non-professionally recorded audio, varying sound quality). As we got to 60m or so of new speaker data, the difference was much less, at least to my non-professional ear. It would be interesting to see (hear?) a detailed analysis of mixing ratios and resulting speaker quality. I wonder if the recording quality of the new speaker has some effect on the benefit, too... |
Beta Was this translation helpful? Give feedback.
-
|
@redoctopus in my case, I'm using spotify podcasts recorded in a studio, and then processed with an audio tool. But there is certainly a distance to a "professional" recording... I also use a smaller subset for HiFiGAN, reaching a point with good quality in the sense of audio quality |
Beta Was this translation helpful? Give feedback.
-
|
@gedefet could you tell me please about your experience - do you have success finetuning FastPitch+Hifigan with a small new dataset duration 15-30 minutes? I tryed to fine-tuning one FastPitch variation but the quality is not good enough and I suspect that fastpitch only gives good quality when training from scratch with large datasets. I would be grateful for any information about your experience. |
Beta Was this translation helpful? Give feedback.
-
|
@gedefet Did you manage to get such the perfect dataset? Could you send me some examples of records from the dataset and generated records trained on it on my email? and how duration of this new dataset did you have when was training? |
Beta Was this translation helpful? Give feedback.
-
|
The quality should be like LJSpeech or HiFiTTS dataset. I recorded it on a professional studio |
Beta Was this translation helpful? Give feedback.
-
|
@gedefet thank you, I understood about quality dataset.. but may be you can sent me examples generated records, it's important for me to know that it sound like human after fine-tuning with small good dataset. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I can't, as they are private, non-public samples |
Beta Was this translation helpful? Give feedback.
-
|
No problem, maybe someone else can answer me.. thanks for the answers! |
Beta Was this translation helpful? Give feedback.
I've updated the FastPitch fine-tuning notebook in #3954, hopefully the procedure for mixing datasets should be more clear.
As mentioned in the tutorial, we don't explicitly have a code section where you can run this mixing in the notebook itself due to resource limitations on Colab. But if you have a local machine to use, you should be able generate a new manifest and re-run the fine-tuning commands with the few command line changes specified without too much trouble (I hope!).