Prompt Cache

[apaz-cli]

When you start the program, it has to load the model, then tokenize the prompt and run it through the model. The time to load the model has been mostly eliminated. Tokenizing is also very fast. It would be nice if it were possible to cache the result of running the prompt through the model and load it on startup as well.

I write long prompts, because it gives better results, but it can take a few minutes for the whole thing to process before the window is responsive. With this change, it should be nearly instant.

[calvintwr]

Would you consider running the interference as part of the bootstrapping process of your app? That should cache the prompts.

[apaz-cli]

@calvintwr This post was made back in April when prompt caching was not supported, but it was implemented a short time after.