How should i properly run the best math llm NuminaMath-7B-TIR?

[JohnClaw]

Link to weights: https://huggingface.co/AI-MO/NuminaMath-7B-TIR
Link to gguf: https://huggingface.co/bartowski/NuminaMath-7B-TIR-GGUF

What is the compatible chat template and llama-cli's command-line options for this great model?

[dspasyuk]

@JohnClaw Prompt template just says to use ### Problem ### Solution keywords, so something like this should work, you might need to change -ngl to lower number if you do not have enough VRAM on you gpu or remove it completely if you do not use GPU at all: ../llama.cpp/llama-cli --model ./NuminaMath-7B-TIR-Q5_K_M.gguf -ngl 31 --log-enable --in-prefix "### Problem:" --in-suffix "### Solution:" -p "What is 500 * 10000?" -c 0 -n 100

[dspasyuk]

You can also run in conversation mode just add -cnv to that, remove -p "What is 500 * 10000?" -n 100 options

[foldl]

Here a full demo:

https://github.com/foldl/chatllm.cpp/blob/master/docs/tool_calling.md#numinamath