Skip to content

Commit 7b5edf1

Browse files
matteoservaggerganovochafik
authored andcommitted
server : support jinja extra template kwargs (Qwen3 enable_thinking feature), from command line and from client (ggml-org#13196)
* initial commit for handling extra template kwargs * enable_thinking and assistant prefill cannot be enabled at the same time * can set chat_template_kwargs in command line * added doc * fixed formatting * add support for extra context in generic template init * coding standard: common/chat.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * coding standard: common/chat.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Apply suggestions from code review coding standard: cosmetic changes Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * fix merge conflict * chat.cpp: simplify calls to apply to ensure systematic propagation of extra_context (+ the odd existing additional_context) * normalize environment variable name * simplify code * prefill cannot be used with thinking models * compatibility with the new reasoning-budget parameter * fix prefill for non thinking models --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Olivier Chafik <olivier.chafik@gmail.com>
1 parent e6c926a commit 7b5edf1

File tree

2 files changed

+4
-0
lines changed

2 files changed

+4
-0
lines changed

common/common.h

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
#include <string>
99
#include <string_view>
1010
#include <vector>
11+
#include <map>
1112
#include <sstream>
1213

1314
#ifdef _WIN32
@@ -381,6 +382,8 @@ struct common_params {
381382
std::string ssl_file_key = ""; // NOLINT
382383
std::string ssl_file_cert = ""; // NOLINT
383384

385+
std::map<std::string, std::string> default_template_kwargs;
386+
384387
// "advanced" endpoints are disabled by default for better security
385388
bool webui = true;
386389
bool endpoint_slots = false;

tools/server/server.cpp

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2110,6 +2110,7 @@ struct server_context {
21102110
/* use_jinja */ params_base.use_jinja,
21112111
/* prefill_assistant */ params_base.prefill_assistant,
21122112
/* reasoning_format */ params_base.reasoning_format,
2113+
/* chat_template_kwargs */ params_base.default_template_kwargs,
21132114
/* common_chat_templates */ chat_templates.get(),
21142115
/* allow_image */ mctx ? mtmd_support_vision(mctx) : false,
21152116
/* allow_audio */ mctx ? mtmd_support_audio (mctx) : false,

0 commit comments

Comments
 (0)