use transfusion implement text to speech

[junphine]

@lucidrains
I've seen e2-tts-pytorch, it is NAR text to speech，And a duration predictor model is required. I want to use transfusion to implement tts, is it feasible?

My implementation method：

hidden_dim = 256

Transfusion(
encoder = vocos.feature_extractor
decoder = vocos.decode
pre_post_transformer_enc_dec = （nn.Conv1d(mel_channels, hidden_dim, 3, 2, 1),
nn.ConvTranspose1d(hidden_dim, mel_channels, 3, 2, 1, output_padding = 1)）
transformer = dict(
dim = hidden_dim,
depth = 8,
dim_head = 64,
heads = 16,
)
)

The first step is to train speech generation using only speech data

The second step uses [text, speech] multimodal data

The current problem is that the loss in the first step did not go down，Is there something missing？

[lucidrains]

@junphine no i don't think so, it should be that simple

when doing speech only training, are you using this script?

[junphine]

@lucidrains Yes，For the most part refer to e2_tts_pytorch.trainer. py
trainning code:

``
for ind in range(self.start_step,self.num_train_steps):
step = ind + 1

        self.model.train()
        if self.accelerator is not None:
            with self.accelerator.accumulate(self.model):
                data = next(dl) # raw audio data, will encode to mel  in model encoder 
                
                velocity_consistency_model = None
                if self.ema_model is not None and self.ema_model.initted:
                    velocity_consistency_model = self.accelerator.unwrap_model(self.ema_model).ema_model


                    loss = self.model.forward(data,modality_type=2,velocity_consistency_ema_model=velocity_consistency_model)

                self.accelerator.backward(loss)

``

current ema_model is not set , so velocity_consistency_model is not used.

the model output loss in range 3 ~ 5 at beginning, after 10000 step still 3 ~ 5. lr is 7.5e-5,batchsize is 8

I will set ema enabled and try agen

[lucidrains]

@junphine which dataset are you using? i'll just setup the training script for you, at least for modality only

[junphine]

@lucidrains Thank you! not public dataset，just some mp3 song，I random choose 1 minute clip as a sample.

[arturlunardi]

@lucidrains +1 this script will be highly appreciated