Setting the synthesizer sentence segmenter for a given language

[polirritmico]

Hi! Great project.

Currently the pySBD segmenter is hardcoded to english:

class Synthesizer(nn.Module):
    def __init__(
        self,
        *,
        tts_checkpoint: str | os.PathLike[Any] | None = None,
        tts_config_path: str | os.PathLike[Any] | None = None,
        tts_speakers_file: str | os.PathLike[Any] | None = None,
        tts_languages_file: str | os.PathLike[Any] | None = None,
        vocoder_checkpoint: str | os.PathLike[Any] | None = None,
        vocoder_config: str | os.PathLike[Any] | None = None,
        encoder_checkpoint: str | os.PathLike[Any] | None = None,
        encoder_config: str | os.PathLike[Any] | None = None,
        vc_checkpoint: str | os.PathLike[Any] | None = None,
        vc_config: str | os.PathLike[Any] | None = None,
        model_dir: str | os.PathLike[Any] | None = None,
        voice_dir: str | os.PathLike[Any] | None = None,
        use_cuda: bool = False,
    ) -> None:
        # etc.
        self.seg = self._get_segmenter("en")

It would be great if that parameter could be set somehow. I don't fully understand the api, so I can't confidently suggest an appropriate approach to achieve this, but maybe following the language arg in the tts_to_file call?

Thanks.

[eginhard]

Yes, if the output of that is not appropriate for your language, the best workaround would be to split the input into sentences yourself and pass them to Coqui separately.

In terms of improvements, the most useful would probably be to also allow passing a user-defined sentence splitter. I'm unlikely to work on this myself though.

[eginhard]

Thank you for merging! I'll try to turn to that underlying synthesizer issue to make sure proper segmenter is always used per language in the coming days.

@teddybear082 Sounds good, here is the issue for that. Maybe send a rough draft or plan first, so I can see the general direction makes sense.

Also, the XTTS model has a completely different sentence splitter, but maybe best to ignore that for now:

coqui-ai-TTS/TTS/tts/layers/xtts/tokenizer.py

Line 43 in 327b23c

def split_sentence(text, lang, text_split_length=250):

[teddybear082]

Also, the XTTS model has a completely different sentence splitter, but maybe best to ignore that for now:

coqui-ai-TTS/TTS/tts/layers/xtts/tokenizer.py

Line 43 in 327b23c

def split_sentence(text, lang, text_split_length=250):

Thanks, today I learned! That's pretty cool it does stuff like fixing abbreviations.

On the overall plan for the coqui-tts project and universal segmenter, are you ok with me adding a language parameter to the Synthesizer split_sentences function and implementing that way or do you need to keep that with just text as the sole parameter?

Basically,

split_into_sentences(self, text) -> list[str]:

Would become

split_into_sentences(self, text, lang = "en") -> list[str]

And I would find the calls to split_into_sentences and pass language.

Otherwise, I would have to find the language when Synthesizer init is called but I don't see how one would know always what the language is at that time.

Or, final option I can think of (but I don't know if it exists) is if any of the variables of self contain the current language if applicable, then can run self.seg = _get_segmenter(self.{variable holding language}) in split_into_sentences.

Just let me know what you think.

[eginhard]

Yes, it's probably not straightforward. There are a few different cases:

1. Multilingual models know the languages they support:

   coqui-ai-TTS/TTS/api.py

   Line 147 in 4c593c6

   return self.synthesizer.tts_model.language_manager.language_names

   And at inference time the language will be specified:

   coqui-ai-TTS/TTS/utils/synthesizer.py

   Line 297 in 4c593c6

   language_name: str = "",

   So you could initialise a segmenter for each of those (changing self.seg to a dict) and call the appropriate one.
2. Loading a monolingual model by name from the API also defines the language, e.g. de for tts_models/de/thorsten/tacotron2-DDC, so that could be passed to the Synthesizer.
3. Loading a monolingual model by path or creating Synthesizer instances directly won't tell you the language, although sometimes it could be inferred from the phoneme_language value in the config if present (for English could be e.g. en, en-us etc.).

It might be useful to add a languages field to the config for all models (i.e. in BaseTTSConfig) and recipes to have that info available, but it wouldn't help with existing models. But that could already be a first PR anyway. XTTS has one:

coqui-ai-TTS/TTS/tts/configs/xtts_config.py

Line 72 in d59aca3

languages: list[str] = field(

[teddybear082]

Would it be worth it to just implement for cases 1 and 2 for now and then fall back to "en"? What do you think about for those cases, just call self.seg = _get_segmenter(lang) [could use try/except to catch unsupported lang, default back to en] at inference time right before

coqui-ai-TTS/TTS/utils/synthesizer.py

Line 333 in 4c593c6

sens = self.split_into_sentences(text)

instead of creating a dictionary of all segmenters? That way I wouldn't have to mess with the core nature of one of the config variables?

[eginhard]

Sure, no need to cover everything right away and English can always be a fallback. Having a dictionary is probably still useful, so the segmenter doesn't get reinitialised at every call. But you don't have to populate it at the beginning, you could update it with new segmenters as needed, then you shouldn't have to touch the config.

[teddybear082]

Great thanks I will work on that and test it out in the coming days.