Help for people with language difficulties: Word completion with multiple choices

[elPapo]

Hi there,

I would like to create a tool to help people (especially children) with specific speech and language difficulties which makes it very difficult for them to write by hand or using a keyboard.

The high-level idea is to provide a tool that would help them by providing (numerous) suggestions for the next word in a very visual way (for children, the words would be filtered through a lexic of safe words and some words would be illustrated). Clicker is a similar tool (but only runs on Windows, and doesn't have all the features I'm considering).

There are many ways to approach this, including Bayesian filtering of the words, etc... but I thought that it sounded like a good fit for llms, as they could capture the context efficiently.

This led me here, I have experimented with a couple of things, and I have so many questions!

I see two different approaches:

• I could enter the beginning of the text several times as different prompts with a relatively high temperature for sampling and hope for the best, as suggested in this discussion.
• I was considering leveraging the logits:
  • I could input the beginning of the text once as a sequence
  • evaluate it
  • extract the logits and sort the token by probability through softmax
  • From each of the top n tokens, I would create new sequences, copying the kv cache and adding the new distinct token
  • then I would evaluate them in parallel to get their next tokens
  • repeating this until I get a token with a space, ponctuation or eog

In the second approach, I could ideally keep track of the probability of the whole sequences of tokens, and repeat the softmax to always keep n sequences, allowing to keep several sequences from the top first token if it leads to different words with high probability~

And that's where I'd really appreciate your help!

1. Does that sound like a reasonable approach?
2. Do you think that could perform reasonably well to run either on a midrange computer or a reasonably priced VPS (probably no GPU, I guess)
3. Can you think of a simpler/better approach?
4. Do you have advices to dimension the different parameters of the model for this use-case? (especially, potentially long promp, but very short output)
5. To find a word, I can't figure out a better way than generating tokens until I find a new token containing a word separator (space, punctuation, eog). Is there a better way?

Thank you very much for reading this far!

[dspasyuk]

@elPapo it is in interesting idea I wanted to incorporate something similar recently in llama.cui using transformers.js it has dedicated framework for this Text-to-Text, take a look https://huggingface.co/tasks/text-generation#completion-generation-models and here https://huggingface.co/docs/transformers.js/en/index