[Frontend][Model] Qwen3Rerank API Server backward compatibility #20239

BetterAndBetterII · 2025-06-30T05:19:56Z

Purpose

This PR adds native support for templated inputs in the /score and /rerank endpoints, enabling models such as Qwen3-Reranker to receive the exact prompt format they expect without client-side preprocessing.

Backward Compatibility

This change is very important because all downstream applications use query+document input instead of instruct input.
This will make it impossible to directly access the qwen3 Reranker model, and some adapter changes must be made.
My change allows vllm to support template input from two aspects, mainly referring to chat_template and chat_template_kwargs:

Opt-in feature: if no score_template is provided anywhere, the control flow is identical to pre-patch behaviour.
- ScoreRequest or RerankRequest can pass in score_template or rerank_template, similar to chat_template
- score_template can also be configured in tokenizer_config.json, which means that in the ideal case, the model comes with its own template. However, the current model does not carry this configuration, which is somewhat different from the ideal. We can use the hf_overrides parameter to make up for it.
Existing clients and models therefore require zero changes.

PR #19260 reports that the current implementation need client side template format.

Changes

Area	Change
protocol.py	Add optional fields score_template and score_template_kwargs to ScoreRequest / RerankRequest.
serving_score.py	① Pick template in priority order: request → hf_overrides → model config.json. ② Render the template and pass the fully-formed string to the tokenizer.
Engine init (hf_overrides)	Allow operators to inject a cluster-wide default template without modifying model artifacts.

Test Plan

New unit tests ensure:
(a) legacy paths remain identical for untreated models;
(b) custom template works via API;
(c) default fallback via hf_overrides succeeds.

Test Result

encountered dependency issue, solving...

(vllm-dev) (vllm-dev) root@873aa6f520cd:/workspaces/vllm# python -c "import torch"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/vllm-dev/lib/python3.11/site-packages/torch/__init__.py", line 409, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: /opt/conda/envs/vllm-dev/lib/python3.11/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkCreate_12_8, version libnvJitLink.so.12

Documentation Update

Add “Templated Score/Rerank” section with curl / Python snippets.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

…ible and support instruction Signed-off-by: BetterAndBetterII <betterandbetterii@gmail.com>

github-actions · 2025-06-30T05:20:03Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Summary of Changes

Hello @BetterAndBetterII, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the flexibility and usability of the /score and /rerank API endpoints by integrating native support for templated inputs. This change is particularly beneficial for models requiring specific prompt formats, as it shifts the prompt formatting responsibility from the client to the server. The implementation prioritizes backward compatibility while offering robust configuration options for template application, streamlining the interaction with diverse model architectures.

Highlights

Templated Input Support: I've introduced native support for templated inputs in the /score and /rerank API endpoints. This allows models like Qwen3-Reranker to receive the exact prompt format they expect directly from the server, eliminating the need for client-side preprocessing.
Flexible Template Configuration: Templates can now be specified in multiple ways: directly within the API request body (score_template/rerank_template), as cluster-wide defaults via hf_overrides during engine initialization, or potentially from the model's tokenizer_config.json (though not currently utilized by Qwen3).
Backward Compatibility: The templating feature is opt-in. If no template is provided, the system behaves identically to its pre-patch state, ensuring existing clients and models remain fully functional without any required changes.
Enhanced Testing: New unit tests have been added to cover various scenarios, including verifying that legacy paths remain unchanged, custom templates provided via the API work as expected, and the hf_overrides fallback mechanism functions correctly, specifically for the Qwen3-Reranker model.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The PR adds templating support for /score and /rerank endpoints. There's a critical issue with inconsistent field names between ScoreRequest and RerankRequest in the protocol, leading to a bug in the serving logic. I've provided suggestions to unify these names. I've also pointed out an area for code improvement in the tests.

gemini-code-assist · 2025-06-30T05:22:09Z

vllm/entrypoints/openai/protocol.py

+    rerank_template: Optional[dict[str, str]] = Field(
+        default=None,
+        description=("A dictionary containing query_template and "
+                     "document_template to format the reranker input.")
+    )
+    rerank_template_kwargs: Optional[dict[str, Any]] = Field(
+        default=None,
+        description=("A dictionary of key-value pairs to be formatted into "
+                     "the rerank model's template.")
+    )


The field names rerank_template and rerank_template_kwargs in RerankRequest are inconsistent with ScoreRequest's score_template and score_template_kwargs. To ensure consistency, unify the field names in RerankRequest to match ScoreRequest.

Also, improve the description for rerank_template_kwargs by using the description from score_template_kwargs.

Suggested change

rerank_template: Optional[dict[str, str]] = Field(

default=None,

description=("A dictionary containing query_template and "

"document_template to format the reranker input.")

)

rerank_template_kwargs: Optional[dict[str, Any]] = Field(

default=None,

description=("A dictionary of key-value pairs to be formatted into "

"the rerank model's template.")

)

score_template: Optional[dict[str, str]] = Field(

default=None,

description=("A dictionary containing query_template and "

"document_template to format the reranker input.")

)

score_template_kwargs: Optional[dict[str, Any]] = Field(

default=None,

description=("Additional keyword args to pass to the template renderer. "

"Will be accessible by the rerank template.")

)

gemini-code-assist · 2025-06-30T05:22:10Z

vllm/entrypoints/openai/serving_score.py

+        template_config = (request.score_template
+                           or self.model_config.hf_config.get(
+                               "score_template"))


The implementation relies on request.score_template, but for a RerankRequest, the field is rerank_template in protocol.py. Fix this in conjunction with the comment on vllm/entrypoints/openai/protocol.py to unify the template-related field names.

gemini-code-assist · 2025-06-30T05:22:10Z

vllm/entrypoints/openai/serving_score.py

+                if request.score_template_kwargs:
+                    context.update(request.score_template_kwargs)


This code relies on request.score_template_kwargs, but for a RerankRequest, the field is rerank_template_kwargs in protocol.py. Fix this in conjunction with the comment on vllm/entrypoints/openai/protocol.py to unify the template-related field names.

gemini-code-assist · 2025-06-30T05:22:10Z

tests/entrypoints/openai/test_score.py

+                "score_template_kwargs": {
+                    "instruction": instruction
+                }


This test uses score_template_kwargs in a /rerank request. Per protocol.py, RerankRequest expects rerank_template_kwargs. Confirm this test aligns with the final protocol definition after the field name unification.

DarkLight1337 · 2025-06-30T05:25:29Z

Is it viable to instead extend the chat template with query/document prompts and enable them via chat_template_kwargs? A bit similar to enable_thinking flag.

DarkLight1337 · 2025-06-30T05:25:50Z

cc @noooop

BetterAndBetterII · 2025-06-30T05:32:54Z

Is it viable to instead extend the chat template with query/document prompts and enable them via chat_template_kwargs? A bit similar to enable_thinking flag.

It is not a direct extension of chat_template, because chat_template actually accepts a message. Here we only want to accept query+document(+kwargs like instruct) and then apply a template. So I think a feasible solution is to propose a score_template or rerank_template to apply, but the specific name needs to be discussed.

DarkLight1337 · 2025-06-30T05:37:21Z

vllm/entrypoints/openai/protocol.py

@@ -1194,7 +1194,16 @@ class ScoreRequest(OpenAIBaseModel):
            "default: 0). Any priority other than 0 will raise an error "
            "if the served model does not use priority scheduling."),
    )
-
+    score_template: Optional[dict[str, str]] = Field(


Ah I see that these endpoints don't use chat templates now. In that case, I prefer separating query_template and document_template into separate arguments to avoid unnecessary nesting.

copy that 🫡

but here we may need to pass in instruction, that is, other parameters may need to be applied to the template. Maybe a template+template kwargs is better?

I think for user convenience, it's better to separate template and template kwargs, since the kwargs aren't used that often

I think template is a triple of query, document, instruction, among which if the user does not give the instruction, use the default instruction.

should basically be consistent with mteb

https://github.com/embeddings-benchmark/mteb/blob/4ff1413316727ecec7ddaeb280fe963f06e7c3fb/mteb/evaluation/evaluators/RetrievalEvaluator.py#L310-L329

This should use the same API for the entire ecosystem, so I think it should be implemented by Sentence Transformers first.

noooop · 2025-06-30T05:44:23Z

Personally，

I think this feature is very important.
I think this feature should be implemented first in Sentence Transformers.
You might submit a pr for Sentence Transformers first.
Then vllm will follow up.

refer to: #19260 (comment)

If you are interested in the score_template, there is also a known issue that you might want to consider.

Template aware prompt truncation to avoid cutting off important instructions.

BetterAndBetterII · 2025-06-30T06:24:22Z

Personally，

I think this feature is very important. I think this feature should be implemented first in Sentence Transformers. You might submit a pr for Sentence Transformers first. Then vllm will follow up.

refer to: #19260 (comment)

If you are interested in the score_template, there is also a known issue that you might want to consider.

Template aware prompt truncation to avoid cutting off important instructions.

Thank you very much. I am very interested in starting the PR of sentence-transformer.
I notice that the original author wants to add prompt and prompt_name to implement the application of templates in CrossEncoder
ref: https://huggingface.co/tomaarsen/Qwen3-Reranker-0.6B-seq-cls/discussions/2#685533e54ea5fb8f10513376

If sentence-transformer supports this, then we need to align this at the OpenAI API server interface level of vllm?

noooop · 2025-06-30T06:36:25Z

If sentence-transformer supports this, then we need to align this at the OpenAI API server interface level of vllm?

You can also leave comments on sentence-transformer and share your thoughts.

aarnphm · 2025-07-02T15:43:40Z

I don't have enough context with rerank atm, but will wait for @DarkLight1337 review.

[Model] Qwen3 OpenAI API Server reranking and scoring backward compat…

0379f79

…ible and support instruction Signed-off-by: BetterAndBetterII <betterandbetterii@gmail.com>

BetterAndBetterII requested review from DarkLight1337, robertgshaw2-redhat, simon-mo and aarnphm as code owners June 30, 2025 05:19

gemini-code-assist bot reviewed Jun 30, 2025

View reviewed changes

mergify bot added frontend qwen Related to Qwen models labels Jun 30, 2025

gemini-code-assist bot reviewed Jun 30, 2025

View reviewed changes

DarkLight1337 reviewed Jun 30, 2025

View reviewed changes

BetterAndBetterII mentioned this pull request Jun 30, 2025

[Feat] CrossEncoder class accept prompt template parameters UKPLab/sentence-transformers#3406

Open

This was referenced Jul 7, 2025

[Model][Last/4] Automatic conversion of CrossEncoding model #19675

Merged

[Bug]: Qwen3 Rerank 模型的准确率存在问题 #20478

Closed

NiuBlibing mentioned this pull request Jul 10, 2025

[Bug]: Qwen3-Reranker-vllm exhibits a large gap between offline and online inference. #20730

Open

1 task

whtDHU mentioned this pull request Jul 10, 2025

qwen3-reranker模型准确率存在问题 QwenLM/Qwen3-Embedding#96

Open

		if request.score_template_kwargs:
		context.update(request.score_template_kwargs)

Uh oh!

[Frontend][Model] Qwen3Rerank API Server backward compatibility #20239

Are you sure you want to change the base?

[Frontend][Model] Qwen3Rerank API Server backward compatibility #20239

Conversation

BetterAndBetterII commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Backward Compatibility

Changes

Test Plan

Test Result

Documentation Update

Essential Elements of an Effective PR Description Checklist

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 commented Jun 30, 2025

Uh oh!

DarkLight1337 commented Jun 30, 2025

Uh oh!

BetterAndBetterII commented Jun 30, 2025

Uh oh!

DarkLight1337 Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

BetterAndBetterII Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

BetterAndBetterII Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

noooop Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

noooop Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

noooop commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BetterAndBetterII commented Jun 30, 2025

Uh oh!

noooop commented Jun 30, 2025

Uh oh!

aarnphm commented Jul 2, 2025

Uh oh!

Uh oh!

BetterAndBetterII commented Jun 30, 2025 •

edited

Loading

noooop commented Jun 30, 2025 •

edited

Loading