Feature Request: add tool calling for deepseek-r1-0528

[0x010010010]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Currently, the llama-server can run the deepseek-r1-0528 unsolth model normally, but when adding the --jinja parameter, the tool calling functionality fails to meet expectations. Despite multiple attempts with the --chat-template-file parameter (e.g., --chat-template-file tool_chat_template_deepseekr1.jinja), it still doesn't work. Could this be an issue with the parsing of the chat-template-file

Motivation

llama-server -m /models/UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf --n-gpu-layers 99 --host 0.0.0.0 --port 9996 --parallel 2 --ctx-size 16384 --cache-type-k q4_0 --api-key sk-123456789 --threads 24 -a DeepSeek-R1-0528-UD-IQ2_XXS --temp 0.6 --top_p 0.95 --min_p 0.01 --seed 3407 --jinja --chat-template-file ~/tool_chat_template_deepseekr1.jinja

Possible Implementation

No response

[danielhanchen]

@0x010010010 When you mean tool calling doesn't work well, could you provide an example if possible?

[0x010010010]

llama.cpp/llama-server -m /models/gguf/UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf --n-gpu-layers 99 --host 0.0.0.0 --port 9996 --parallel 2 --ctx-size 16384 --cache-type-k q4_0 --threads 24 -a DeepSeek-R1-0528-UD-IQ2_XXS --jinja --temp 0.6 --top_p 0.95 --min_p 0.01 --seed 3407

try:
response = client.chat.completions.create(
model="DeepSeek-R1-0528-UD-IQ2_XXS",
messages=[
{"role": "user", "content": "Provide today's weather for New York and London."}
],
tools=tools
)
except Exception as e:
print(f"Error details: {e.response.json()}")

response.json()

'{"id":"chatcmpl-xUdXjRHQeJ1jb1Alx8raMZCpWFKZAHcU","choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"I need to retrieve the weather for both New York and London. Since the tool only allows querying one city at a time, I'll make two separate function calls. Here are the tool calls:\n\n\n<｜tool▁calls▁begin｜>\nWe are asked to provide today's weather for New York and London.\n However, note that the tool only allows querying one city at a time and the function name is \"get_weather\".\n Also, the description says that for Chinese cities we need to use the English name. For non-Chinese cities, we can use the standard names.\n Since New York and London are not in China, we can use their standard names.\n\n But note: The function only takes one city at a time. So we need to call the function twice.\n\n However, the user's request is for both cities. We must call the function for each city separately.\n\n Let's break it down:\n\n Step 1: Call the function for New York.\n Step 2: Call the function for London.\n\n But note: The assistant can only call one tool at a time? Actually, we can make multiple calls in the same response? \n However, the format requires one tool call per block. So we can do:\n\n We can write two tool calls in the same response? Actually, the format allows multiple tool calls by having multiple blocks.\n\n According to the format, we can do:\n\n function:get_weather\\n {\\"loc\\": \\"New York\\"}\\n \n\n and then\n\n function:get_weather\\n {\\"loc\\": \\"London\\"}\\n \n\n But note: the assistant's response should be in the format that the system expects. Since we are only allowed to issue tool calls in the specified format, and we can have multiple such blocks.\n\n However, the example format shows only one block. But in practice, we can have multiple tool calls by having multiple blocks.\n\n Let's do:\n\n First, for New York:\nfunction:get_weather\\n{\\"loc\\": \\"New York\\"}\\n\n\nThen, for London:\nfunction:get_weather\\n{\\"loc\\": \\"London\\"}\\n\n\nBut note: the assistant must issue both tool calls in the same response? The problem says \"When a tool call is needed, you MUST use the following format\". It doesn't say only one per response.\n\n However, we are only allowed to use the function when we need to. Since we need two queries, we should issue two calls.\n\n But wait: the tool description says \"Only one city name can be entered at a time.\", meaning that each call can only have one city.\n\n So we have to make two separate calls.\n\n Therefore, I will generate two tool calls.\n\n Important: The JSON must be valid.\n\n Let's write the response accordingly.\n\nI need to retrieve the weather for both New York and London. Since the tool only allows querying one city at a time, I'll make two separate function calls. Here are the tool calls:\n\n<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\njson\\n{\\"loc\\": \\"New York\\"}\\n<｜tool▁call▁end｜>\n<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\njson\\n{\\"loc\\": \\"London\\"}\\n<｜tool▁call▁end｜>","refusal":null,"role":"assistant","annotations":null,"audio":null,"function_call":null,"tool_calls":null,"reasoning_content":"We are asked to provide today's weather for New York and London.\n However, note that the tool only allows querying one city at a time and the function name is \"get_weather\".\n Also, the description says that for Chinese cities we need to use the English name. For non-Chinese cities, we can use the standard names.\n Since New York and London are not in China, we can use their standard names.\n\n But note: The function only takes one city at a time. So we need to call the function twice.\n\n However, the user's request is for both cities. We must call the function for each city separately.\n\n Let's break it down:\n\n Step 1: Call the function for New York.\n Step 2: Call the function for London.\n\n But note: The assistant can only call one tool at a time? Actually, we can make multiple calls in the same response? \n However, the format requires one tool call per block. So we can do:\n\n We can write two tool calls in the same response? Actually, the format allows multiple tool calls by having multiple blocks.\n\n According to the format, we can do:\n\n function:get_weather\\n {\\"loc\\": \\"New York\\"}\\n \n\n and then\n\n function:get_weather\\n {\\"loc\\": \\"London\\"}\\n \n\n But note: the assistant's response should be in the format that the system expects. Since we are only allowed to issue tool calls in the specified format, and we can have multiple such blocks.\n\n However, the example format shows only one block. But in practice, we can have multiple tool calls by having multiple blocks.\n\n Let's do:\n\n First, for New York:\nfunction:get_weather\\n{\\"loc\\": \\"New York\\"}\\n\n\nThen, for London:\nfunction:get_weather\\n{\\"loc\\": \\"London\\"}\\n\n\nBut note: the assistant must issue both tool calls in the same response? The problem says \"When a tool call is needed, you MUST use the following format\". It doesn't say only one per response.\n\n However, we are only allowed to use the function when we need to. Since we need two queries, we should issue two calls.\n\n But wait: the tool description says \"Only one city name can be entered at a time.\", meaning that each call can only have one city.\n\n So we have to make two separate calls.\n\n Therefore, I will generate two tool calls.\n\n Important: The JSON must be valid.\n\n Let's write the response accordingly."}}],"created":1752457921,"model":"DeepSeek-R1-0528-UD-IQ2_XXS","object":"chat.completion","service_tier":null,"system_fingerprint":"b5797-de569441","usage":{"completion_tokens":585,"prompt_tokens":202,"total_tokens":787,"completion_tokens_details":null,"prompt_tokens_details":null},"timings":{"prompt_n":202,"prompt_ms":1585.901,"prompt_per_token_ms":7.8509950495049505,"prompt_per_second":127.37238957538963,"predicted_n":585,"predicted_ms":28252.126,"predicted_per_token_ms":48.29423247863248,"predicted_per_second":20.70640630726339}}'

[danielhanchen]

@0x010010010 Could you remove --chat-template-file ~/tool_chat_template_deepseekr1.jinja - the upload we did already has an inherent tool calling chat template which we fixed

[0x010010010]

It seems the problem hasn't been resolved yet.

llama.cpp/llama-server -m /models/UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf --n-gpu-layers 99 --host 0.0.0.0 --port 9996 --parallel 2 --ctx-size 16384 --cache-type-k q4_0 --api-key sk-123456789 --threads 24 -a DeepSeek-R1-0528-UD-IQ2_XXS --temp 0.6 --top_p 0.95 --min_p 0.01 --seed 3407 --jinja