[Frontend][Model] Qwen3Rerank API Server backward compatibility

[BetterAndBetterII]

Purpose

This PR adds native support for templated inputs in the /score and /rerank endpoints, enabling models such as Qwen3-Reranker to receive the exact prompt format they expect without client-side preprocessing.

Backward Compatibility

This change is very important because all downstream applications use query+document input instead of instruct input.
This will make it impossible to directly access the qwen3 Reranker model, and some adapter changes must be made.
My change allows vllm to support template input from two aspects, mainly referring to chat_template and chat_template_kwargs:

• Opt-in feature: if no score_template is provided anywhere, the control flow is identical to pre-patch behaviour.
  • ScoreRequest or RerankRequest can pass in score_template or rerank_template, similar to chat_template
  • score_template can also be configured in tokenizer_config.json, which means that in the ideal case, the model comes with its own template. However, the current model does not carry this configuration, which is somewhat different from the ideal. We can use the hf_overrides parameter to make up for it.
• Existing clients and models therefore require zero changes.

PR #19260 reports that the current implementation need client side template format.

Changes

Area	Change
protocol.py	Add optional fields score_template and score_template_kwargs to ScoreRequest / RerankRequest.
serving_score.py	① Pick template in priority order: request → hf_overrides → model config.json. ② Render the template and pass the fully-formed string to the tokenizer.
Engine init (hf_overrides)	Allow operators to inject a cluster-wide default template without modifying model artifacts.

Test Plan

New unit tests ensure:
(a) legacy paths remain identical for untreated models;
(b) custom template works via API;
(c) default fallback via hf_overrides succeeds.

Test Result

encountered dependency issue, solving...

(vllm-dev) (vllm-dev) root@873aa6f520cd:/workspaces/vllm# python -c "import torch"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/vllm-dev/lib/python3.11/site-packages/torch/__init__.py", line 409, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: /opt/conda/envs/vllm-dev/lib/python3.11/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkCreate_12_8, version libnvJitLink.so.12

Documentation Update

Add “Templated Score/Rerank” section with curl / Python snippets.

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Summary of Changes

Hello @BetterAndBetterII, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the vLLM API by introducing native support for templated inputs in the /score and /rerank endpoints. The primary goal is to ensure backward compatibility and direct usability for models like Qwen3-Reranker, which require specific prompt formats, by allowing the server to handle the templating internally. This change streamlines the integration of such models by removing the need for client-side prompt preprocessing.

Highlights

• Templated Input Support: Introduced score_template and score_template_kwargs to ScoreRequest, and rerank_template and rerank_template_kwargs to RerankRequest in the API protocol. This allows clients to specify how query and document pairs should be formatted for models that require specific input structures.
• Flexible Template Resolution: Implemented a priority order for applying templates: first, templates provided directly in the API request are used; if not present, hf_overrides (allowing cluster-wide defaults) are checked; finally, the model's config.json is consulted. This provides multiple ways to configure templating.
• Qwen3-Reranker Compatibility: Specifically enabled native support for the Qwen3-Reranker model, which expects templated inputs. This change allows the model to be directly accessed and used within vLLM without requiring client-side preprocessing of prompts.
• New Preprocessing Logic: Refactored the input processing for scoring and reranking by introducing a _preprocess_score method. This new method centralizes the logic for applying templates to query and document pairs before they are tokenized and sent to the model.
• Comprehensive Testing: Added new unit tests to validate the templating functionality, covering scenarios with custom templates provided via the API, default fallbacks configured through hf_overrides, and ensuring that existing, untemplated paths remain unaffected and backward compatible.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces templated inputs for the /score and /rerank endpoints. The implementation is well-structured, and the tests are comprehensive. However, there's an inconsistency between the RerankRequest protocol definition and its usage in the implementation and tests, which could lead to runtime errors. There are also opportunities to improve clarity in the template rendering logic.

[gemini-code-assist]

The prefix and suffix strings are hardcoded here and also in the server fixture on lines 83-84. To improve maintainability and avoid duplication, define them as module-level constants and reuse them in both places.

[gemini-code-assist]

The logic for applying the template can be simplified for better readability. A clearer approach would be to get the template string and then format it only if it exists.

                query_template_str = template_config.get("query_template")
                doc_template_str = template_config.get("document_template")

                formatted_t1 = query_template_str.format(
                    **context) if query_template_str is not None else t1
                formatted_t2 = doc_template_str.format(
                    **context) if doc_template_str is not None else t2
                return formatted_t1, formatted_t2