[Feature request] Auralis optimisations of XTTS

[erew123]

🚀 Feature Description

Hi @eginhard. Hope you are keeping well!

Its erew123 from AllTalk

Someone has pointed this out to me: https://www.astramind.ai/post/auralis and I think this is the GitHub https://github.com/astramind-ai/Auralis

Its a little beyond my pay grade, but maybe its of interest to the Coqui scripts. I dont know if you have seen this, or the author is posting on here with you, but I thought you may like to see it:

Firing their document into AI for a quick "Here is what they claim"

---

The author claims to have optimized XTTS-v2, a text-to-speech model, making it faster, more resource-efficient, asynchronous, and safer for production environments. Here are the key points and the performance gains:

What They Did

1. Understanding the Code and Challenges:

   • Overcame a lack of prior experience in audio tech.
   • Debugged and worked around outdated dependencies and repos.

2. Tokenizer Optimization:

   • Replaced a custom tokenizer with a Hugging Face-compatible FastPreTrainedTokenizer.
   • Improved token splitting logic to maintain audio quality while handling memory-efficient truncation.

3. Model Reorganization:

   • Refactored the original architecture, which used GPT-2-like models and a HiFi-GAN vocoder, to eliminate unnecessary computations during inference.
   • Optimized the HiFi-GAN component to use in-place operations, drastically reducing memory usage.

4. Integration of vLLM for GPT-2:

   • Overcame challenges in adapting vLLM for multimodal GPT-2, including token cache management and continuous batching.
   • Addressed vLLM's limitations on repetition penalties and hidden state collection, customizing its behavior for audio-specific tasks.

5. Asynchronous Execution:

   • Made components non-blocking using asyncio.

6. Optimized Workflow:

   • Avoided redundant token and embedding calculations during iterative decoding.
   • Adapted position ID tracking to align with unique conditioning inputs for multimodal tasks.

Performance Gains

1. Speed:

   • Leveraging vLLM and deduplicating computations significantly reduced inference time.

2. Resource Efficiency:

   • Memory consumption was slashed by optimizing HiFi-GAN for inference.
   • Reduced overhead by restructuring the GPT-2 and conditioning modules.

3. Production Suitability:

   • Ensured asynchronous, non-blocking execution for smoother integration into UI frameworks like Pulsar.
   • Increased safety by moving from .pth to safer formats and handling positional encoding appropriately.

4. Accessibility:

   • Made the enhancements available to the open-source community for broader adoption.

The overall result is a production-ready, optimized XTTS-v2 that is significantly faster and more memory-efficient, with asynchronous capabilities enabling smoother integration into applications.

---

Thanks Erew123

[eginhard]

Thanks for sharing! I wasn't aware of it. Will take a look.

[erew123]

@eginhard FYI the requirements they have bump Pytorch to 2.5.1... I have no idea if they are actually using something from that version of PyTorch. Just so you are aware. Thought it interesting though!

Thanks

[Paladinium]

Is there evidence that the improvements Auralis describes are indeed significant?

[HussainBadreddeen]

Any updates on this? @eginhard

i just came across this as im seeing inference time of around 4 seconds for a 153 char sentence in Arabic:

XTTS Internal (Sentence): Tokenization took 3.08ms
The following generation flags are not valid and may be ignored: ['temperature', 'top_p']. Set TRANSFORMERS_VERBOSITY=info for more details.
XTTS Internal (Sentence): GPT generate took 4106.11ms
XTTS Internal (Sentence): GPT forward (latent) took 27.87ms
XTTS Internal (Sentence): HiFi-GAN took 70.98ms
XTTS Internal (Sentence): Total sentence processing took 4214.18ms

XTTS Internal: Overall inference method took 4214.19ms
TTS inference for the text took 4214.59ms.

this line XTTS Internal (Sentence): GPT generate took 4106.11ms points to the gpt model.
Im not that experienced in running something like vLLM for speeding this up, but i could give it a go as i need inference to be a bit faster for realtime

[eginhard]

Im not that experienced in running something like vLLM for speeding this up, but i could give it a go as i need inference to be a bit faster for realtime

Sure, feel free to test that implementation and let me know if any of their improvements could easily be added to Coqui.