Is it possible to use a custom made voice models for speech generation (TTS) in flutter?

[mzramle]

I recently a pytorch model with an .index metadata using Applio voice training. Then i use another voice changer for exporting the model .pth and .index into an ONNX model. The question is i want to use the ONNX voice model for flutter TTS using the sherpa tts engine. Is it possible?

[csukuangfj]

Is it a VITS model?

For instance, if you use piper to train your model, it is possible to support it in sherpa-onnx.

We support all models from piper.

If not, please tell us which framework you are using.

[csukuangfj]

Is your model open-sourced?

[mzramle]

open-sourced? yeah maybe. Its just i don't know how to use it for my flutter application. Also i want to know how to implement sherpa-onnx TTS engine because from your documentation its in chinese so i could not understand it (https://www.hovthen.com/SherpaOnnxTtsEngineAndroid.html). But when i go to this link it (https://github.com/jing332/SherpaOnnxTtsEngineAndroid) I'm not sure where to implement the TTS engine because im not sure whether it is a flutter dart based language or not. It is also a whole complete android app which i cannot see where to fit my use case of integrating sherpa-onnx TTS engine reading from ONNX model.

[csukuangfj]

How do you convert text to tokens for your model?

If you use espeak-ng, then once you convert your model to sherpa-onnx, then all code in sherpa-onnx can be reused.

---

i know how to use native android for tts engine
and also know how to use flutter to develop an android tts app (we will add doc for this)

However, we have no experience with how to use flutter to develop an android tts engine.

Note android tts app is not equal to android tts engine.

[mzramle]

oh i dont know any conversions for text to tokens. I just want to know if the custom onnx model can be used in this sherpa-onnx tts by way of using the custom onnx as model reference. I assume that the rest will be done by sherpa-onnx library itself.

alright for a simpler question, disregarding the android tts app and android tts engine, I just want to know can i just use my onnx model as reference for sherpa-onnx inference and then use maybe something like any tts provided by sherpa-onnx or any other compatible tts?

Also do i have to convert my onnx to 'sherpa-onnx'? Because having .onnx format is enough right no need for further conversion?

[csukuangfj]

I just want to know can i just use my onnx model as reference

First, you need to make sure the model is VITS and it can be exported to onnx

Second and the last one, you have to find out how the model gets its input.

If you only have the model but don't know how to feed inputs to it, then you cannot do anything with the model.

[avinoam134]

hey, does the flutter package support offloading the models to Neural Engine / GPU?

[csukuangfj]

it supports nvidia gpu

[avinoam134]

and is it any different than the NE/GPU support from the native android languages (Java/Kotlin)? If not, is it inherently the case (like if behind the scenes, the Flutter package calls the native platforms to deal with inference and so automatically receive any hardware accelaration benefiets that the native implementations do)

[csukuangfj]

Actually, it is an issue of onnxruntime. I suggest that you ask in the onnxruntime github repo.

We use onnxruntime inside.