audiobook in TTS

[teleshoes]

buggins#353
@plotn i developed a feature i love that you might be interested in.
it would need a lot of work to make it plug-and-play. the NLP magic is currently done mostly outside of coolreader

[teleshoes]

NOTE: this is a working audiobook impl, but it is FAR from polished, and it is NOT plug-and-play. i am already using it all the time, though, so i figured i'd stick it here in case someone else is interested.

FEATURES

• supports reading from mp3/ogg/flac/wav files, instead of android TTS engines, inside the TTS ReadAloud Module
• supports sentence navigation, pause/play/next/prev, etc
• moves the selected sentence as playback progresses, and plays from the selected sentence
• does not assume audiobook and ebook are identical, makes best guesses and moves along
  • even large missing or additional sections generally work fine
    • if the audiobook plays a section missing from the ebook, the visual sentence selector waits for the audio to catch up
    • if the ebook has a section missing from the audiobook, the visual sentence selector skips a sentence every half second until it catches up.
• does not skip over audiobook intros/music etc. the full audiobook is played, and the place in the book moves along while the content is read

USAGE

• requires creating a .wordtiming text file for each e-book OUTSIDE OF COOLREADER
  • this file can be generated from any speech-to-text system like vosk that supports per-word timings
  • ebook-audiobook-words is a script for generating this using vosk, pandoc, gnu-diff, python, + perl
  • see:
    • https://github.com/teleshoes/wolke-home-config/blob/e140268/bin/ebook-audiobook-words
    • https://github.com/teleshoes/wolke-home-config/blob/e140268/bin/vosk-words-json
    • https://github.com/alphacep/vosk-api
    • https://pypi.org/project/vosk/
• once created, place the wordtiming file, ebook, and audiobook mp3/flac/etc files in the same directory
• open the ebook as normal, and start TTS. if there is a wordtiming file, the audiobook is used

EXAMPLE

• attached is A Christmas Carol, by Charles Dickens, from Project Gutenberg and Librivox (all works in the public domain)
  • https://drive.google.com/drive/folders/1abepyfOW9on94tiZpoEdN4QBYuud_jk4?usp=sharing
  • includes: e-book (.txt), audiobook, (.flac), and wordtiming file, (*.wordtiming)
• download all files, and extract the zip file to any directory on the device
• wordtiming file was generated with this script:

  ebook-audiobook-words \
    a_christmas_carol_charles_dickens_project_gutenberg_19337.txt \
    A_Christmas_Carol*.flac \
    -o a_christmas_carol_charles_dickens_project_gutenberg_19337.wordtiming
  

[plotn]

buggins#353 @plotn i developed a feature i love that you might be interested in. it would need a lot of work to make it plug-and-play. the NLP magic is currently done mostly outside of coolreader

Hola! Say please (using simple english words, what has happened, and what should be (if we...) . I'm trying to go go back to this planet

[teleshoes]

i made a feature in coolreader.
it makes the ReadAloud button read from an MP3 audiobook, instead of google-tts.
it uses artificial intelligence (vosk) to match them together, so you can read along with an audiobook.

it is not easy-to-use, and you have to do the calculation in a separate program for each audio book, but it works well.

the main benefit is that google-tts sounds like a robot that hates you, mispronounces words, and is generally not a good experience.
audiobooks are generally higher quality, contain expression, pronounce names correctly, and have extras like theme music and multiple voice actors.

this lets you get the same benefit of coolreader TTS/ReadAloud, but with a much better audio experience.

[teleshoes]

as for what you should do, i am just letting you know that this exists.

it should be easy enough to port my feature to KnownReader, but i still only use CoolReader, unfortunately. i will not be porting it unless i switch to KnownReader. however, someone else could port this easily, and all of the code i wrote for it is GPLv2+.

[plotn]

Thank you, i'm sorry i just a little bit drunk. KnownReader always been a proof of concept project and it is. Could you show you code? I thought about it thinking about yandex speech engine, but ut was too much expensive those days

[teleshoes]

code is linked above, buggins#353

[teleshoes]

the speech-to-text part is external. that is in:
https://github.com/teleshoes/wolke-home-config/blob/e140268/bin/ebook-audiobook-words
https://github.com/teleshoes/wolke-home-config/blob/e140268/bin/vosk-words-json

[plotn]

Took a detailed look. It's really great, I think.
But I need detailed guidance how to make whole process (I will investigate it myself, sure).

Is it works somehow as follows?:

1. We have "ordinary" book, e.g. fb2;
2. We have an audiobook for it, it can be not exactly match (by content), but it is similar;
3. We make word-timing for this audiobook via external took, like your perl script calling vosk engine;
4. We join this artifact all together (in the ebook reading app);
5. We won! - we can read book by text, book by audio and we do not listen to mechanical voice.

Questions:

1. Dont you mind if I implement your fearure inside KR? I saw your code, it is possible?
2. Can I pesent your ideas at KR forum and find someone who also interested?

[teleshoes]

yes, that is correct!

• my script ALSO needs the FB2 file. it outputs a best-guess position in the audio file for each word in the FB2 file
• the artifact output of my script is a CSV text file that is read by coolreader
• coolreader still has to do a lot of work, though, to line up coolreader's sentence model against the list of words

1. yes, by all means! the code is GPLv2+, so you must comply with that license. knownreader is already GPLv2+. if you wind up using my code, attribute me. (be careful if you bundle some software into coolreader to do the artificial intelligence work. vosk is apache licensed, so thats fine)
2. by all means

ideas that are NOT implemented:

• replace pandoc in my script with code from coolreader

  • my script uses pandoc to process the FB2 file into text, and coolreader does not work EXACTLY the same as pandoc, though it is closed
    • my script allows the audiobook and the ebook to be VERY different and still work perfectly (~75% accuracy is usually enough), but the ebook parsing in my script and the ebook parsing in coolreader MUST be extremely similar
  • if i replaced pandoc in my script with, say coolreader --run-headless --give-list-of-words --ebook=file.fb2, then we could use the exact same code, and it would always be the same word list
  • this would mean modifying coolreader/knownreader to have a command line interface for processing ebooks for my script

• fix the coolreader TTS sentence model to be predictable and calculated in advance

  • the vast majority of the startup time in coolreader is spent just analyzing the ebook to get a list of sentences that TTS uses
    • the code currently navigates to the start, and does selectNextSentence() repeatedly, until reaching the end
    • each sentence gets an (x,y) point from the selection, and that point is what i use to get which sentence is selected
    • if we replace the model to be a list of sentences, and generate the (x,y) points on the fly, it will improve performance enormously
    • if we add a command line interface to coolreader/knownreader for this, too, then we can do ALL the computational work could be done outside of coolreader, and the startup would be very fast

[plotn]

I did it in KR!!!!
Magnficient!

[teleshoes]

hooray! i added a cache yesterday for the slow part of generating the sentenceinfo. im going to look into ways of maybe making a command line interface to generate that cache file on a laptop. that may mean a rewrite of some of the java code in c++, unfortunately. i dont know when ill get around to that.

im also looking at improving the alignment of the sentenceinfo with the wordtiming. also, sometimes large books crash after starting/stopping TTS a few times, so i may have a large memory leak.

[plotn]

I got the crashes, possibly it is because you too often call ensureAudioBookPlaying();
I did some workaround, take a look

[plotn]

while I did not got the knowledge of wordtiming creation - could you please convert some classical russian books?
Master and Magarita (Bulgakov)
War and Peace (Tolstoy)
Crime and punishment, Idiot (Dostoevsky)

to present to the community, sure. Guess these books are widely accessible.

[teleshoes]

haha, i did all 3 of those in english already and they worked well. on consideration, cyrillic will probably break my script, and maybe this pull request. ill look when i get a chance

[plotn]

Give me them please? in english is ok

[teleshoes]

unfortunately, tolstoy and dostoevsky are too large to upload, and the master is enjoying a nice shady place in USA copyright limbo, at least for the next 50 years. (all of the translations ive used are probably copyrighted separately, anyway)

i can, however, assist you with running the script, if you like, and then you can show whoever wants to maintain this crazy idea of mine. :)

[plotn]

Unfortunately even first example works buggy (maybe I did not acuurate code move). When I play from the begginig, it is ok, when I start from 38 of 377 page - it fails

[teleshoes]

oh, yes i can reproduce that. i may have introduced that yesterday, with the changes made to identify sentences by their DOM position, instead of (x,y) coordinates, so that i could cache them without regard to font size.

i find that a christmas carol works correctly up until Out upon merry Christmas! What's Christmas, and then it breaks right after, at time to you but a time for paying bills without money. the wordtimings look correct, so the problem must be in the sentence matcher in coolreader. i reproduced it in another book as well.

[teleshoes]

@plotn found and fixed :)
(it was ' quote characters. difference in parsing by pandoc vs coolreader. this concern will disappear if i can expose coolreader as the tool to generate the wordtimings input file, instead of pandoc)

[plotn]

(cristmas carol)

[plotn]

I guess the following:

1. Cr/kr generate "template" wordtiming file
2. Some other tool enriches this file by real timings.

Or not?
And what we do now?

[teleshoes]

there are THREE things that must be aligned:

1. word in audio book
2. word in ebook
3. selected sentence (physical location of word in ebook)

matching 1+2 is done in the perl script, and works beautifully. that is called wordtiming, and the result is the time of each EBOOK word.

matching that result, wordtiming, to (3), the sentence in coolreader, is done in coolreader, because only coolreader can predict how coolreader will parse the ebook into a DOM and break up sentences. this is the part that had a bug i fixed today. if i could use cr directly from the perl script, i could do ALL the work up front, and do both matchings at once.

[plotn]

Okay, lets go a little bit slower. E.g.:

1. we have audiobook with sound "yesterday all my troubles was so far away, now it looks as thought they here to stay";
2. We have text (or fb2) book with text "yesterday all my troubles was NOT so far away, now it SEEMS as thought they here to stay". (a little bit different);

go further. We take 1) and parse sound to wordtiming:
00:00,yesterday
00:01,all
00:03,by
00:10,troubles
00:11,was
....

This is done via AI and perl, yes?

what is next? when we apply textual book?

(I'm sorry, I'm nerd)
:))))

[teleshoes]

perl

• P1) vosk-timing-data - run vosk-words-json on WAV files, get statistics on each word
  • output is a big JSON file
  • this is the only CPU-intensive/long-running step
  • the result is LZMA compressed and cached
  • the cache can be generated with or without an ebook, for processing later
  • (i plan on running this on all audiobooks as i acquire them)
• P2) audio-word-timing - process vosk-timing-data into a CSV with three columns: AUDIO_WORD,START_TIME_SECONDS,AUDIO_FILE
  • this is the start time of each word in the AUDIOBOOK
• P3) audio-word-list - make a copy of audio-word-timing and remove the START_TIME_SECONDS column
• P4) ebook-word-list - process the EPUB/FB2/TXT file into a list of words (one word per line) with pandoc
  • this step tries to apply the same rules the coolreader will use later
• P4) ebook-audio-diff - align audio-word-list and ebook-word-list
  • this is the ONLY STEP that compares the ebook to the audiobook
  • this step handles bad spelling, bad pronunciation, proper nouns, missing passages, extra words, skipped sentences, footnotes read in line instead of at the end of the chapter, EVERYTHING that is different between the audiobook and the ebook
  • it uses the myers difference algorithm, finding the Longest-Common-Subsequence
  • i.e.: its literally just diff -y
• P5) ebook-word-timing - combine ebook-word-list, ebook-audio-diff, and audio-word-timing to get ebook timing
  • take ebook-word-list and add two column, START_TIME_SECONDS and AUDIO_FILE
  • using ebook-audio-diff, find the highest index of the word in audio-word-list that is not part of the longest-common-subsequence of a later word
  • take that index and get the timing from audio-word-timing, and fill in START_TIME_SECONDS/AUDIO_FILE columns with it
  • after this point, the words in the audiobook are not used and never appear again
    • you will never see mispronounced words, spoken errors, etc. every word from here on out appears, in order, in the FB2/EPUB/TXT
  • this is the start time of each word in the EBOOK
  • this is the contents of *.wordtiming, and is the final output of the perl script

coolreader

• CR1) sentence-info - when audiobook-tts starts, navigate to each sentence and get info
  • jump to page 0
  • select the first sentence on the page
  • select the next sentence, repeatedly, until there are no sentences left in the book
  • after selecting each sentence, record the sentence-text and the dom-start-pos
  • this is done in CPP, invoked from a JNI file, and sent to java as a List<SentenceInfo`
  • this is turned into an (improper) CSV with two columns, START_POS and TEXT
    • start pos is a DOM id, from ldomXPointerEx->toString() (it never has any commas)
    • TEXT is allowed to contain commas, because its the last column (hence, this is an improper CSV)
    • e.g.: /text/p[45].135, having little or no money in my pocket
  • java caches this sentence info in a file, *.sentenceinfo, if coolreader has write perms where the ebook is
  • this step is the only slow part in coolreader
  • it is purely the coolreader sentence structure parsing
  • it has NOTHING to do with audiobooks, or wordtimings, or anything
• CR2) sentence-words - parse each sentence into a list of words, as close to step P4) as possible
  • its never exactly the same, because pandoc is not coolreader, but the difference is SMALL
• CR3) sentence-start-times - compare sentence-words to ebook-word-timing file
  • load *.wordtiming, parse into a list of word/start-time pairs
  • for each sentence, take each word in that sentence and try to apply to the next word in wordtiming
  • a sentence must match EVERY SINGLE WORD in sentence-words to a word in wordtiming
  • however, allow skipping up to 20 WORDTIMING words (this is arbitrary, but if its very long, passages could be skipped)
  • if a sentence does not match every word (happens all the time), consume the words that did match, use the lowest start time matched, and continue to the next sentence
  • the output here is a start TIME and start PHYSICAL-POSITION for every sentence (this is the final pre-processing goal)
• CR4) start-playback - play audiobook instead of TTS
  • when tts-start is invoked, get the DOM location of the start of the initially selected sentence
  • select the closest sentence in sentence-start-times
  • open the audio file, and seek to the start time
  • NOTE: if this is the first sentence in the audio file, ALWAYS seek to 0
    • this way, you hear the full audiobook
    • start music, copyright, "Start of CD Number Three", etc
• CR5) continue-playback - select the next sentence as audiobook position continues
  • do not stop playback, ever, without user interaction (this way, you get to hear the end of each audiobook file)
  • when media playback time is after the next sentence in sentence-start-times, select the next sentence as if user clicked Next >>
  • when media playback STOPS, and the next sentence is the FIRST sentence of a new audio file, start the next audio file

[teleshoes]

• P1-P5 is done externally in ebook-audiobook-words perl script, using vosk and pandoc
• CR1 should be done externally, as it is quite slow, but cannot, as it requires the EXACT algorithm coolreader uses to parse sentences
• i want to make a CLI version of coolreader that does step CR1
• CR2-CR3 could also happen externally, if we had CR1 done. this would leave just playback to android

[teleshoes]

TL;DR no, the output of perl is NOT just the audiobook. it is the timing of the words in the EBOOK (fb2/epub/txt)
(therefore, it will say "yesterday all my troubles was NOT so far away, now it SEEMS as thought they here to stay)

so, the output, would be something like this (with the audiofile omitted, and #comments added), assuming each word takes 100ms to say.

yesterday,0.0
all,0.1
my,0.2
troubles,0.3
was,0.4
not,0.4 #will be same as previous word, also will be lowercase
so, 0.5
far, 0.6
away, 0.7
now, 0.8
it, 0.9
seems, 1.0 #will be at the same time as 'looks' would have been
as, 1.1
thought, 1.2
they, 1.3
here, 1.4
to, 1.5
stay, 1.6

[plotn]

Thank you for such detailed guidance!

[teleshoes]

aaand it was fairly easy to make a CLI to generate *.sentenceinfo (step CR1) i did it inside cr3qt, although it doesnt actually use anything in QT in particular, and it never shows the QT app window. (it was just a convenient starting base)

(its not in the PR yet, needs some cleanup)

[teleshoes]

@plotn i moved the main perl script to its own repo:
https://github.com/teleshoes/ebook-audiobook-wordtiming