Specify token budget during inference

[vladfaust]

How can we tell a decode-only model a "token budget" when running inference loop so it doesn't just stop mid-way once the limit is reached, but somehow "plans ahead" to fit the response into the budget? Thanks for any tips.

[dspasyuk]

@vladfaust What type of decode-only model are you working with? For general use you can do something like this:
../llama.cpp/llama-cli -ngl 10 -m ../../models/qwen2-7b-instruct-q2_k.gguf -p "<|im_start|>system\nAnswer as a pirate<|im_end|>\n<|im_start|>user\nTell me a story<|im_end|>\n<|im_start|>assistant" Depending on the model your prompt template will change.

[vladfaust]

I want to limit the response length. Usually I tell the model something like "keep the response short (1-2 sentences)", but it's not reliable. Instead, I'd like to be able to set the token budget explicitly, e.g. to 50 tokens; with that information, a model would somehow "look ahead" during inference, picking the tokens which would result in fitting the budget.

Also, I'm embedding Llama.cpp into a C++ project, so I'm more interested in the programmatic side rather than CLI usage. Thanks for the response, though.