Adding non-tokens to transcription

[manjuke]

Hi All,
I would like to add some special symbols in to transcription like lang. id of each utterance "<english>" and retrain the model. Is there a way to do that? Can I do it by passing some parameters to process_asr_text_tokenizer.py.
Currently, I have tried adding some unused symbols like "<" for English, and ">" for Hindi etc. Pls let me know if this is correct? Or let me know if there is any better/efficient way to do that. Thanks

[manjuke]

@titu1994 @jbalam-nv Any suggestion on this ? Thanks

[titu1994]

There's no way to update a prebuilt Sentencepiece Tokenizer. You can only create a new one.

Special tokens encoded as single Unicode token (< or >) is fine and a valid approach. However if you try to do it for every single token you will not be able to train a CTC model.

CTC and RNNT are both alignment losses, not classification task. They align timesteps between two sequences.

These kinds of extra marks have no acoustic info, so even if the encoder has global attention it is unlikely you will get perfect output pairs.

Next up, CTC has the T > U requirement (T being audio seq length, U being token seq length). By putting double the tokens to be emitted, you have an even higher chance of violating this requirement and your loss will be inf + grad for such samples is 0 so you won't learn anything.

Now, RNNT may be able to do this. It doesn't have this restriction as CTC, and has an implicit LM within it. You could try it maybe.

[manjuke]

Thanks for the detailed answer @titu1994.
Instead of single Unicode tokens like "<" or ">", is it possible to have some some special tokens like <english>, <hindi> etc. These special tokens are kind of similar to <pad> & <unk> tokens that are supported by Nemo's process_asr_text_tokenizer.py script for building tokenizers.

I am fine with building custom Sentencepiece Tokenizer. Currently, I use Nemo's process_asr_text_tokenizer.py script to create a custom Sentencepiece Tokenizer. Hope this is correct. If there is any other alternate approach or if it's incorrect, please suggest the same.

Currently, if I there are tokens/symbols like <english>, <hindi> in my transcription, they are getting tokenized into English tokens like "e, n, g, l, i, s, h, d, i, <, >". So, finally the original tokens/symbols in the transcription like <english> <hindi> would have been lost.

My exact issue lies at on "how to build custom Sentencepiece Tokenizer," which tells that the symbols like <english>, <hindi> are unique and special tokens, and should not be treated as transcription, & sub-tokenize these special tokens

I completely agree that these extra marks will not have any acoustic info, and T > U requirement to be satisfied. But, I am adding only one tag per utterance, irrespective of total no. of languages, which is fine I believe.

I have already implemented it with special symbols likes "<" ">" with CTC. But, I wanted to have it implemented using special tokens like "<english> or <hindi>" etc. Because, scaling to more tags (around10 tags) will be difficult with single Unicode tokens approach.

Thanks a lot

[titu1994]

I can look into how to add Sentencepiece special tokens during Tokenizer building but at the moment there is no support for it.

[manjuke]

Sure, Thanks. Please let me know, in case if it gets implemented.