Writing Prediction with GPT-2 model

[CelsiaSolaraStarflare]

Is it possible for the program to get the text that the user is on and then also show a grey(hint) word in front of the sentence so that the user can try to finish their sentence by using the AI predictions? We can use the GPT-2 model or some of the pretrained llama or alpaca models for the prediction.
(To be honest, I am only more familiar with pygame and tkinter because we use them constantly in school).

[SchBenedikt]

In fact, I've thought about this before, which is why I added the OpenAI Playground integration. I could imagine this with a new chatGPT model like GPT-3.5, since currently everyone has access to it via the API key. But if you do this via OpenAI's API key, you have to keep recharging, which in turn costs something. What I would like even better would be Code-LLama, but I don't know what the API key is. It might be convenient to run Code-LLama locally on the device via the application, but that would cost a lot of performance, which I don't think is great. However, one option would be to use a free API key. In my opinion, perhaps the best option would be if one could somehow integrate Microsoft Copilot or GitHub Copilot, which has always provided suggestions in VS Code. And now you can also answer that Github Copilot costs something. As a rule, there are monthly costs. But that wouldn't be the case if you verified yourself as a student. In addition, my software already has a very extensive integration with Github, which is why I consider Github Copilot almost the best.
What do you think about it?

[SchBenedikt]

Maybe you can also program your own prediction with Tensorflow?

[CelsiaSolaraStarflare]

I think we can try to use the Llama or GPT-2.0 fine tuned version model. It is not as good as 3.5 and is only capable of finishing and continuing the sentence(instead of Q&A), but it can be used without internet(run locally). Although it requires some bit of CPU power but the lightweight model I guess will be fine for some CPU like intel i7 10th Gen, 16Gb or so RAM. Because I was in a deep learning class in school so we made some sentence continuers like these before(not super hard just that I don't know how to add them onto the user interface).
\\\\\\\\\\\\
Microsoft Copilot: I think Copilot is on Microsoft Azure, there is like a free tier for 200 something queries per month, but it needs an registered account on Microsoft Azure which needs a credit card. So I think not everyone could use the API(considering students may not have a card).
Github Copilot: I cannot find the way to use it in the code for right now, I might try it like next month; but good point.

[SchBenedikt]

Yes, if it runs locally, that's of course even better. If you know of any code that can help us, then maybe just open your own repository or share the code with us so that we can see how we can embed it.

[CelsiaSolaraStarflare]

Falcon7BC.py.zip
This is the Falcon super-lightweight model(based on Fine tuned GPT)'s program. Right now I have set the max_new_tokens to five, so that it does not generate a lot of content(which is quite slow). I estimated it for about two to three seconds per token and a token is a single word. I think it will auto download the model(around 14Gb) or you have to git clone part of the program from huggingface. You can set the prompt to ":"+the text edit and then use the output for ":"+content.

Of course this is quite large as it is fine tuned to be capable like early builds of GPT3.0; another method is just changing the model.load to 'gpt2' and from transformers import gpt2, which will create a new blank model that is usable but better with finetuning. It is around 200Mb when first created, and you can allow the model to train on the content the user writes in the text-editor once they are saved(by training in another thread) and using time.sleep in the thread to decrease CPU usage power and train more slowly for older computers.

[CelsiaSolaraStarflare]

(and the output in terminal looks like this without the l1 l2 stuff because I altered the base code)

[CelsiaSolaraStarflare]

Although it is unable to correct spelling mistakes like "phthalocyanine"

[CelsiaSolaraStarflare]

Oh wait but I think this could be something that the Text Editor is different and exclusive from other normal editors. We can try to use the fine tune technology based on the already base trained model in the text so that the predictor trains based on the writing style of the person(maybe another function on the tool bar). Also maybe an text style store feature where people can upload their model styles or download others. (If we want to do this then I think we can use LoRA for GPT on a common base model which I haven't tried before but could be more efficient than the full model).