Chat with Code in vim

[VJHack]

It would be cool to see a chat feature implemented in Vim so that the user can ask questions about their codebase. This is inspired by many of the leading code editors like Cursor.ai, Github Copilot, and Windsurf.

A simple example of how this what this would look like:

This would increase the utility of this tool by making better use of llama-server for coding assistance.

I'm aware that this project is focused on 'Local LLM-assisted text completion'. Does this feature fall within the scope of this project?

[VJHack]

This falls under one of the secondary objectives in the "p1 : LLM-based code completion engine at the edge 1" announcement. Dropping the link here for visibility.

ggml-org/p1#1 (comment)

[ggerganov]

IMO a full-blown chat interface would be out-of-scope for this particular project, because the complexity of the implementation would increase dramatically - we need to support things like visualizing and accepting code diffs, printing formatted code blocks, extra windows, etc.

There are 2 possible approaches:

• Extend llama.vim with very specific features, such as "refactor", "explain", "generate", etc. This set of features can be configured to hit a separate endpoint that would serve a larger, non-FIM LLM. Most of the existing FIM functionality should be reusable. Probably we only need to add streaming support somehow, since these tasks would typically require more time to complete.

• Implement a second plugin that supports a more complicated chat-like UI.

[VJHack]

Agreed. I suggest we go with the first approach and limit the scope of the chat interface by only offering a few select commands as you've mentioned. But before getting into that, since FIM autocomplete has been demonstrated in llama.vim, I think it would be better to bring llama.vscode up to speed and integrate this feature for VS Code before attempting it with vim. The reason being that there’s better UX support for VS Code with View Containers, Quick Picks, Notifications, and more, significantly reducing the complexity of the implementation.

This set of features can be configured to hit a separate endpoint that would serve a larger, non-FIM LLM

If we are having to serve a non-FIM LLM along with a FIM LLM, the user would need to serve multiple models across different servers if I’m understanding this correctly. This seems like it would get quite messy. Could this potentially be done with same model? I'll need to investigate this further.

[ggerganov]

But before getting into that, since FIM autocomplete has been demonstrated in llama.vim, I think it would be better to bring llama.vscode up to speed and integrate this feature for VS Code before attempting it with vim. The reason being that there’s better UX support for VS Code with View Containers, Quick Picks, Notifications, and more, significantly reducing the complexity of the implementation.

I agree.

If we are having to serve a non-FIM LLM along with a FIM LLM, the user would need to serve multiple models across different servers if I’m understanding this correctly. This seems like it would get quite messy. Could this potentially be done with same model? I'll need to investigate this further.

The upcoming feature for extracting LoRA adapters from fine-tuned models would make it possible to use a single server to server both the FIM and non-FIM models:

ggml-org/llama.cpp#11131

However, I think having 2 separate servers would still be beneficial because we would have to maintain 2 separate KV caches - one for each model. The reason is because the FIM and non-FIM models have different prompt formats.

[JosefAlbers]

I've actually been working on something similar - split-window LLM responses with keyboard shortcuts for prompting and code insertion. If it helps with your implementation decisions or saves you some work, check it out: VimLM. Happy to answer questions about the approach I took for the UI. Just thought it might be relevant to what you're discussing.

[MartyLake]

I have been using https://github.com/madox2/vim-ai before I discovered that llama.vim existed. It has what you wish for I think, see AIEdit and AIChat in general.