Input truncation is automatic on ollama you need to add a single flag to fix it

[devlux76]

Which version of the app are you using?

latest head

Which API Provider are you using?

Ollama

Which Model are you using?

All of them

What happened?

Thanks for an excellent product. I love that it integrates with so many providers, but the ollama integration is bugged due to an incredibly short sighted design decision of the ollama team that I don't think you're aware of.

time=2025-03-04T02:30:04.818-07:00 level=WARN source=runner.go:129 msg="truncating input prompt" limit=2048 prompt=16101 keep=4 new=2048

Ollama by default truncates input to 2048 tokens this is really quite tiny.

In order to override this all you need to do is to set the limit in the message options to something reasonable. There is no reason not to use the model's context length - new for this.

"options": {
    "num_ctx": 128000
  }

This can of course cause resource issues, so best practice is to measure what you current have, plus the size of the expected generation and then cap it at the model's max context size. This way you don't waste a bunch of resources when you only need a much smaller context at the moment.

Source: https://github.com/ollama/ollama/blob/main/docs/faq.md

Steps to reproduce

Relevant API REQUEST output

Additional context

No response

[devlux76]

Just a quick update. On a hunch I tried the "openai compatible" provider, but it's the same story...

time=2025-03-04T03:15:35.453-07:00 level=WARN source=runner.go:129 msg="truncating input prompt" limit=2048 prompt=16101 keep=4 new=2048

[hannesrudolph]

Can you please provide me a specific model so I can download it and try to repro it? Also can you provide your model config settings screenshot and a screenshot of the chat session where it fails?

EDIT: Sorry I had a swear word in there I did not mean to include. I should read what I type!. Sorry @devlux76

[mrubens]

I know what you mean @devlux76 and I think it would help a lot with the ollama integration so you don't have to manually create a new model every time (i.e. remove step 3 from https://docs.roocode.com/providers/ollama#setting-up-ollama). Let me convert this to a feature request and we can keep discussing.