[Q/FR] "Legacy" Completion

[NightMachinery]

Is there a way to do so-called legacy completions?

• https://platform.openai.com/docs/guides/text-generation/completions-api
• https://platform.openai.com/docs/api-reference/completions/create

res = openrouter_client.completions.create(
    model="mistralai/mixtral-8x22b",
    prompt="""...""",
    stream=True,
    echo=False, #: Echo back the prompt in addition to the completion
    max_tokens=100,
)

This API is lower-level than the ChatML API, and usable with base models, which can be quite strong as an auto-completer.

Related:

• [Q] How do I use a base (completion only, no instruct/chat) model? s-kostyaev/ellama#108 (comment)

[ahyatt]

It's certain a possibility, thanks for raising this. One issue is that it isn't super well supported by the commercial companies, which are focused on chat. But I'll take a look and see how feasible it is.

[NightMachinery]

@ahyatt The legacy completion API also supports fill-in-the-middle completions, using the suffix argument for supplying the suffix that comes after the completion.

Here is an example:

prompt = "def say_hello("
suffix = """):
  print('hi', name)"""

response = client.completions.create(
    model="gpt-3.5-turbo-instruct",
    prompt=prompt,
    suffix=suffix,
    max_tokens=10
)
# response.choices[0].text should be name

This API is great for an autocomplete model. Unfortunately, OpenAI is the only cloud API for FIM completion that I could find. OpenRouter does not seem to support the suffix argument. OpenAI also doesn't support it for GPT4.

[NightMachinery]

I did some research on FIM support. Some open models also support FIM, though each uses its own prompt format to do so. E.g.,

• Code Llama 2
  • Structured FIM API

<PRE> {prompt_prefix} <SUF>{prompt_suffix} <MID>

• DeepSeek

<｜fim▁begin｜>def quick_sort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[0]
    left = []
    right = []
<｜fim▁hole｜>
        if arr[i] < pivot:
            left.append(arr[i])
        else:
            right.append(arr[i])
    return quick_sort(left) + [pivot] + quick_sort(right)<｜fim▁end｜>

• StarCoder
  • API

<fim_prefix>before <fim_suffix> after<fim_middle>

Right now, gpt-3.5-turbo-instruct still seems to be the best FIM model available. I did not look for unofficial FIM finetunes of the stronger open models such as Llama 3 or Mixtral.

I suggest adding support for gpt-3.5-turbo-instruct, as it is straightforward and strong. For the other models, I am unsure if emacs is a good place to maintain their FIM prompt template such that the end user uses a single FIM API. Perhaps this should be done by ollama.

Note that normal completion (without suffix) is supported by tons of models on OpenAI-compatible servers such as Together, Fireworks, etc. So that should be supported by this package, regardless of FIM.

[NightMachinery]

I found an open issue on Ollama about abstracting the FIM API:

• API for FIM tasks ollama/ollama#3869

[ahyatt]

This is great info, thanks. Having to do per-model work is a bit tricky because in GPT4All, we don't even know what model is being run. I'm not sure how we can handle that at the moment. Barring that problem, it seems possible but messy. I think supporting normal completion seems more likely, but that's also very model-dependent, and so has the same issues as GPT4All.

[NightMachinery]

@ahyatt Thanks. Why are normal completions model dependent? Even chat-tuned models are internally run as next token predictors with a hardcoded prompt template, so the completion API should be possible for all local models, and some cloud models. The only model dependency is whether the cloud supports this API, but if it doesn't, it will give a 404 error anyway, so it could be effectively treated as if all models support the API.

For FIM, the ollama API above uses the underlying FIM API from llama.cpp, so that also might not be model-dependent. Of course, gpt-3.5-turbo-instruct does not run on llama.cpp, so the FIM API might need some model-dependency.

Does this elisp package use GPT4All under the hood?

[ahyatt]

My understanding, perhaps flawed: some models are these raw foundation models, which basically operate as completion models. But many models are chat models (most of the popular models are like this), tuned to give complete responses, not designed to be used for completion. The results would be strange; I don't think you'd get an actual completion. So to use a completion API you have to use it with only models that are designed to be used that way.

And about GPT4All, yes, it's one of the options in the llm package, so we should if possible design things to support this package as well. But it's the only package we support that gives no control to the caller of the llm package about what model is being used.

[NightMachinery]

@ahyatt Yes, chat-tuned models probably aren't that good for completion, but they can be used anyway. I don't think exposing the lower-level completion API (which can be used with an appropriate template to produce the higher-level chat responses) has any downsides at all. It allows more tinkering and extensibility, if anything.

Given a sufficiently long context, these chat-tuned models might even work as good completers. The chat RLHF lobotimizes the models, but when the context grows, the model returns back to its roots of next token prediction.

[ahyatt]

I think we'll have to see how it works in practice. I'll try to look into in the next few weeks, thanks for bringing it up!

[NightMachinery]

@ahyatt Mistral’s new 22B code model also supports infilling:

• https://mistral.ai/news/codestral/
• https://huggingface.co/mistralai/Codestral-22B-v0.1#fill-in-the-middle-fim

This is usable both locally and via the API.

[ahyatt]

Thanks for the information. I've been taking a look at this, and the support is pretty spotty. I think the basic problem is that although the ability to do completions is how the foundation models work, companies like Open AI can't release the completion functionality from them due to lack of trust/safety filtering on the outputs. It is available in a deprecated API, but I'm not sure that's something that we want to depend on.

AFAICT, here's the current state of completion functionality:

Open AI: Prefix & infill completion, but with legacy API.
Claude: Prefix completion API, not legacy.
Gemini: Nothing
Ollama: Prefix completion API, not legacy. Actual ability to do completions as completions varies by model.

That's not bad, I think we could conceivably implement this. However, I still wonder if it is necessary. You can get the same functionality by using the existing chat functionality. For example:

(llm-chat ash/llm-openai (llm-make-chat-prompt "Return the contents of <INFILL>.  When I got home that morning, I noticed something was missing.  <INFILL>  That's when I knew I had been visited by clowns.  INFILL="))
"my collection of colorful balloons had vanished, and there was confetti scattered all over the floor."

I'd first want to know if it really is unworkeable to use it via the chat interface this way. It doesn't seem particularly clunky to do this.

[NightMachinery]

@ahyatt I actually have a chat-based implementation of infilling. It's slow, complex, heuristic-based, expensive, error-prone and nowhere near as good as GH Copilot. I have both of them bound to different hotkeys, and I almost always use GH Copilot. And GH Copilot is shit on my internet connection and often doesn't return any results. In those cases, I just give up. The chat-based solution is just not worth it, it's easier to type the code manually.

I am not saying it's impossible to work around the problem with more prompt engineering, but it's fighting an uphill battle for no real gain. We also need strong models for the chat based solution to work at all, while even a weak auto-regressive model should still be useful with the completion API.

ClosedAI et al. are indeed not likely to keep providing us with a completion API, and their current offerings aren't that good anyway. But open models should already be good enough. Specifically, this new Mistral Codestral should be quite good. DeepSeek Coder was also pretty good when I checked it out, though I only used the chat version manually.

[NightMachinery]

I have heard that CodeGemma 2B is also a good autocomplete model. This model is so small that it doesn’t even have a chat-tuned version.

• https://huggingface.co/google/codegemma-2b-GGUF

[ahyatt]

I've started a completion branch, and made a basic ollama implementation. I've tried the CodeGemma model, but the quality wasn't very good.

I'm trying the following

(llm-complete-async (make-llm-ollama :completion-model "codegemma:latest")
                    "(defun fibonacci (n) " (lambda (x) (save-excursion (goto-char (point-max))
                                                                        (insert x)))
                    #'ignore)

Like I feared, the result is more chat-like than completion-like. It may be possible to fix this by choosing just the right model. Also, the completion API in Ollama has no way to add a suffix, unless possibly by playing around with the template, which is something I don't know enough about.

Please feel free to try it out and let me know if you can get a good quality answer out of it with various ollama models.

[NightMachinery]

Thanks.

Ollama's infilling API has not been merged.

I implemented Mistral's FIM API myself. Testing it, it's not too bad, but it's not very good. So I guess you're right for now. Mistral's FIM seems even worse than my hacked together chat-based solution when using GPT4O.

My biggest problem with my hacky chat-based solution is how it doesn't return the exact part asked for. E.g., when asking to complete:

prefix: print("Hel
suffix:)

it might return world"), while the correct answer is lo World".

Unfortunately, Codestral FIM API also has this problem.

I guess the Codestral API might be worth it becuse of cost, as they offer both pay-as-you-go and a monthly subscription for unlimited (with minutely rate limits) API usage.