Writing Godot extension for llama.cpp

[hazelnutcloud]

Hi everyone,

I've been writing a little extension for the Godot game engine to run LLM inference using llama.cpp.

So far, I have everything set up following the simple code example. However, when I try to run my project using my extension, every time I try to run some completions in the same context instance, it would always seem to use the very first prompt that I input.

Here's what my code looks like

I've been scratching my head trying to solve this problem for days now. Would really appreciate any help at all in trying to solve this. Thanks!

[dranger003]

Before your first llama_decode try adding llama_kv_cache_clear(ctx).

[hazelnutcloud]

That worked! thank you so much <3

[Jeff-intcode]

4 short questions simple to answer:

1- Do you plan to make a video tutorial explaining how to install and use it?

2- What uses does it have inside or outside the game? I mean can it be used to dialogue with an NPC (or something like that), have the AI control an NPC, or is it just like a Github copilot for GDscript?

3- Do you have to download Llama 3 or another LLM to use it offline or is it via API?

4- What was your inspiration to create this extension and for what purpose did you create it?

extra- is it finished?