Why is FishSpeech API Faster Than My Self-Hosted Setup?

[RomaricLocuta]

Hi everyone,
I'm trying to understand the performance gap between inference with the FishSpeech API and my self-hosted deployment Here's my configuration:

Technical Details:

• Hardware: NVIDIA H100 GPU.
• Self-hosted setup: 10 parallel server instances, load-balanced.
• Client request strategy: Limited to 5 concurrent requests using a semaphore.
• Inference optimization: Using --compile and testing both with and without --half precision, but neither impacts the observed performance.

Observations:

As the attached graph shows, the FishSpeech API is faster at processing the 89 requests (maximum 5 requests sent at the same time) than my local setup.

Questions:

1. Does the API leverage additional optimizations not included in the self-hosted code?
2. Are there specific configurations or best practices for maximizing performance on the H100 GPU ?

Thanks for any insights!

[TetoDola]

same issue