Idea: Pre-processing text with an LLM before audiobook generation

[hebbihebb]

Hey there Deniz and team,

I’ve been having a great time using Abogen and wanted to share how it fits into my little audiobook production chain.

My main use case is taking webnovel EPUBs, converting them to Markdown, and then turning them into TTS-ready markdown files. Before sending them to Abogen, I run the text through a small side app I built called TTS-Proof github.com/hebbihebb/tts-proof. It uses a local LLM to clean up the text for spelling, grammar, and TTS safety, fixing things like stylized letters or awkward capitalization that can confuse voices such as Kokoro or XTTS.

After the cleanup, I pass the corrected Markdown to Abogen, which handles the audio generation beautifully. That workflow made me wonder if a similar LLM preprocessing step might make sense as an optional feature inside Abogen itself. It could automatically tidy up rough EPUBs, fanfics, or webnovel text before TTS synthesis.

This isn’t a feature request, just an idea to think about. If Abogen eventually adds something like this internally, my helper app would be redundant, and I’d honestly be happy to see that happen.

Thanks for all your work on Abogen. The Markdown to audiobook pipeline has become incredibly smooth lately.

— Hebbi

[OpaTsupa000]

Can this help with processing .PDF's? I have quite a list of unfixable PDFs that I would like to convert to Audibooks. Everything I tried to clean up the text failed and results are unusable.

On the subject of cleaning up EPUBs,
Most of my books are non-fiction with many footnotes that I don't wish to hear in the audio. I use Calibre and the help of ChatGPT to remove all footnotes from the text. It's just a few steps, but it would be nice if it was automatized.

Many thanks to Abogen project in general.
I tried them all and this is the best tool right now.

[hebbihebb]

Fixing extracted text from PDFs is possible, but it depends on what exactly is wrong with it. I have not tested any PDFs yet.

The constant editing of EPUBs is what made me decide to build an app for this in the first place. I have not tested automatic removal of notes yet, but it could be possible.

[denizsafak]

@hebbihebb It's a great idea, and I'd love to integrate it, but Abogen is currently close to 6GB. If the model file were around 2GB, the total size would increase significantly. Furthermore, the local server, LM Studio, and Node.js requirements make things difficult. It would be more convenient if we could import it directly into Abogen, for example, by "pip install tts-proof", as we do with Kokoro.

Apart from this, Abogen converts PDF, EPUB and MD files into plain text files before processing them, meaning the model only needs to correct these plain text files.

So the large file sizes, processing times, and compatibility issues are what got me thinking.

[Suiyou]

I was doing something similar with a little python app that allowed me to preprocess an epub or txt to check every word for mispellings, proper names, contraptions, stutters and the like that TTS programs have issues. I especially needed something to ensure the pronunciation of names that are not of English origin, for example japanese like [Basen] (/'ba''se''n'/) or spanish names within an predominantly english text because Kokoro allows to process the phonemes. The ideal thing would be to have an LLM preprocess the text consulting the user how to modify the text and generate a dictionary or something similar so the next time it knows how to proceed without user input. And one last thing that it would be great to have is a way to automatically asign a voice to each character for abogen/Kokoro to generate the corresponding audio. @hebbihebb I would check your app, it seems interesting.

[hebbihebb]

Yeah, that makes sense, @denizsafak

After working on TTS-Proof and now SATCN, I’ve realized that building this stuff into Abogen would probably be a huge job anyway. The size isn’t really the issue, it’s all the pieces that have to work together to clean text properly.

Honestly, I was just hoping someone smarter than me would get into improving TTS text quality so I wouldn’t have to keep doing it myself. After a few weeks of working on it, I’ve realized this is a tougher problem than I first guessed.

So it wasn’t a real feature request, just tossing the idea out there in case it sparks something later.
Thanks again for keeping Abogen moving forward, love the app.

[denizsafak]

@hebbihebb tts-proof and satcn both seem super handy. Text cleanup is a big part of the process, so tools like these could make a huge difference. I’ll explore how they could be integrated, but even if that’s not possible right away, just using them alongside could save a ton of time. Appreciate the great work!