Key not present in Dictionary: "content"

[sheldonth]

Hi

Running a local instance with the recommended model and when I run the plug everytime I go into insert mode I see the following series of errors.

Thanks

[n2noon]

I accidentally got this when I didn't have enough memory for it to run, does your llama-server look something like this?

error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
llama_decode: failed to decode, ret = -3
srv  update_slots: failed to decode the batch: KV cache is full - try increasing it via the context size, i = 0, n_batch = 1024, ret = -3`

[autonome]

I was experiencing this, and disabled all other vim plugins and it fixed the problem.

Then I re-enabled each plugin one by one, testing each time.

And it worked every time.

No idea.

[haunt98]

I use lazy.nvim to load llama.vim lazily by trigger command LlamaEnable. Then I got same error:

[jhonnyV-V]

I'm having the same problem in neovim with the lazy plugin manager, but something weird happens, if I execute the :Lazy command and update any plugin the error stops, suddenly llama.vim just start working

[DavidLarsKetch]

I'm seeing the same error in neovim after a few minutes of use. This is the error in my llama-server logs while it is processing a prompt:

init: sequence 0 does not start from the last position stored in the memory
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
srv  update_slots: Invalid input batch., i = 0, n_batch = 1024, ret = -1
slot      release: id  0 | task 447 | stop processing: n_past = 4803, truncated = 0
srv    send_error: task id = 447, error: Invalid input batch.

I am not seeing out-of-memory issues, as mentioned in other reports.

I've disabled all plugins and the errors showed up.

Here's my llama-server call:

llama-server -m path/to/model --port 8012 -ngl 99 -fa -ub 1024 -b 1024 -dt 0.1 --ctx-size 0 --cache-reuse 256

[rockerBOO]

I'm also getting init: sequence 0 does not start from the last position stored in the memory. Updating the plugin or restarting the client don't seem to help with this but need to restart the llama-server. Then it works fine again. Might be some issue with the llama-server. I'm using a pretty new version to the current date.

[pnb]

I also get sequence 0 does not start from the last position stored in the memory occasionally, which is the main source of this issue for me. Usually somehow triggering a refresh of some sort will fix it, e.g., by opening another file (triggering a new, different request to llama-server) and then going back. There are perhaps many different ways to trigger changes to the request made to llama-server to avoid the problematic request, and thus lots of different things that will temporarily fix it.

There might be multiple issues at play, such as possibly llama-server issues or maybe just bad requests from the plugin, but at the very least there are a lot of reasons why a request might fail (internet blips when running not on localhost, for example) so the plugin could probably be improved by handling them gracefully.

It is easy hard to replicate this exact issue, but easy to generate similar errors from unexpected server responses by setting the endpoint to something silly, such as :let g:llama_config['endpoint'] = 'https://example.com'

[rockerBOO]

I was thinking and poking at it some more and it seems to happen more often when I change my cursor window to a floating window (but could be other windows?). Maybe if I do it while it would be getting completions it might be a component of it. Makes sense but it is hard to have it be very consistent. I think ideally it would fail a bit better because it interrupts input every keypress after it starts to fail. Right now me fast track to getting it working again is restart the text editor completely (maybe LlamaStart/LlamaStop would be faster). I am running on localhost if that is a component.

[DavidLarsKetch]

Testing this out currently: https://github.com/DavidLarsKetch/llama.vim/commit/42de4c4f14149d0afd6cb4b66bcbccab338a7b63

So far so good...

[pnb]

@DavidLarsKetch I tried something like that as well but I realized it has the disadvantage that invalid responses still end up in the cache, potentially evicting valid responses and reducing the probability of cache hits. It also seems like it would not work in case the server responds with invalid JSON (e.g., over a flaky network proxy) where json_decode will show errors.

I tried something in #73 instead that checks the response before it gets added to the cache.