[Bug]: Qwen3 Rerank 模型的准确率存在问题

[MrBlue-1996]

Your current environment

按照官网下载最新版的vllm pip包，daily的，0.9.2rc版本 main分支 GPU卡为 H20

🐛 Describe the bug

启动命令： python3 -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8181 --served-model-name Qwen3-Rerank --model /mnt/data/.t1/dianjin-0701/Qwen3-Reranker-0.6B --task score --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}' &> start.log & curl命令： curl http://127.0.0.1:8181/v1/rerank \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen3-Rerank", "query": "什么是机器学习？", "documents": [ "机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式", "机器学习是一种编程语言，用于开发网站", "机器学习是数据库管理系统的一种", "机器学习是操作系统的一种类型" ], "top_n": 2 }' 结果：

使用bge-rerank-v2-m3版本模型进行同样测试
启动命令：
python3 -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 8181 --served-model-name bge-rerank --model /mnt/data/.t1/dianjin-0701/bge-reranker-v2-m3/ --task score &> start.log &
curl结果：

很明显Qwen3-rerank-0.6B模型在目前的使用中有问题，谁能帮我解决这个问题

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @noooop

[MrBlue-1996]

@noooop 能帮我看看这个问题吗

[MrBlue-1996]

类似的测试基于Qwen3-rerank已经转化好的seq-cls模型 tomaarsen/Qwen3-Reranker-0.6B-seq-cls

[noooop]

qwen3_reranker using the converted model and the official model should yield almost the same results

whether it is pingpong test or end-to-end testing

https://github.com/noooop/snippet/blob/main/converting2SequenceClassification/test_qwen3_reranker.py

The official model dtype is bf16, so vllm uses bf16 inference. The tomaarsen/Qwen3-Reranker-0.6B-seq-cls dtype is fp32, and vllm uses fp16 inference.

You can try using fp32, but statistically there is not much difference between fp16, bf16 and fp32.

refer to #19344

[MrBlue-1996]

qwen3_reranker using the converted model and the official model should yield almost the same resultsqwen3_reranker 使用转换后的模型和官方模型应该会产生几乎相同的结果

whether it is pingpong test or end-to-end testing无论是乒乓测试还是端到端测试

You can try using fp32, but statistically there is not much difference between fp16 and fp32.您可以尝试使用 fp32，但从统计上讲，fp16 和 fp32 之间没有太大区别。

https://github.com/noooop/snippet/blob/main/converting2SequenceClassification/test_qwen3_reranker.py

@noooop sorry to bother,but What I mean is that there is currently a problem with the accuracy rating of the qwen3 rerank model's responses. The question I asked is, what is machine learning? His correlation rating for the following four answers is clearly incorrect
you can see this picture

the highest relevance score should be Machine learning is a branch of artificial intelligence that enables computers to learn patterns from data through algorithms ,but the model rate this answer last place

[noooop]

正在本地尝试复现这个问题

[MrBlue-1996]

正在本地尝试复现这个问题

感谢！具体环境如下

[noooop]

首先

• Qwen3 Reranker

Caution

Please use the query_template and document_template to format the query and document for better reranker results.
without template, the results are almost as random. PTAL #19344

你一定要用 query_template and document_template

请参考 #19260

[noooop]

如果都不使用 query_template and document_template 结果几乎没有差别

curl http://127.0.0.1:8000/v1/rerank \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tomaarsen/Qwen3-Reranker-0.6B-seq-cls",
    "query": "什么是机器学习？",
    "documents": [ "机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式", "机器学习是一种编程语言，用于开发网站", "机器学习是数据库管理系统的一种", "机器学习是操作系统的一种类型" ]
  }'

vllm serve tomaarsen/Qwen3-Reranker-0.6B-seq-cls 

{"id":"rerank-4f53115e17674498a3a7317fa664ee11","model":"tomaarsen/Qwen3-Reranker-0.6B-seq-cls","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一
  类型"},"relevance_score":0.984752357006073},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9655764698982239},{"index":1,"document":{"text":"机器学习
  一种编程语言，用于开发网站"},"relevance_score":0.9000037312507629},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.783326268196106}]}


vllm serve tomaarsen/Qwen3-Reranker-0.6B-seq-cls --dtype float32


{"id":"rerank-68484c2c25494d67810d4ab6e49f26b2","model":"tomaarsen/Qwen3-Reranker-0.6B-seq-cls","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一
  类型"},"relevance_score":0.9847448468208313},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9657185673713684},{"index":1,"document":{"text":"机器学习
  一种编程语言，用于开发网站"},"relevance_score":0.9001880884170532},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.7831804156303406}]}




vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'



  curl http://127.0.0.1:8000/v1/rerank \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-Reranker-0.6B",
    "query": "什么是机器学习？",
    "documents": [ "机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式", "机器学习是一种编程语言，用于开发网站", "机器学习是数据库管理系统的一种", "机器学习是操作系统的一种类型" ]
  }'


vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'


  {"id":"rerank-44ace860b82f4d61aef0eedf4e4a763c","model":"Qwen/Qwen3-Reranker-0.6B","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一种类型"},"relevance_score":0.9840936064720154},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9648551344871521},{"index":1,"document":{"text":"机器学习是一种编程语
  ，用于开发网站"},"relevance_score":0.9005298018455505},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.7853085398674011}]}

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}' --dtype float32


{"id":"rerank-bbf8be7557734ff0904430b39a63fe4e","model":"Qwen/Qwen3-Reranker-0.6B","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一种类型"},"relevance_score":0.9847448468208313},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9657185673713684},{"index":1,"document":{"text":"机器学习是一种编程语
  ，用于开发网站"},"relevance_score":0.9001880884170532},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.7831804156303406}]}

[MrBlue-1996]

如果都不使用 query_template and document_template 结果几乎没有差别

curl http://127.0.0.1:8000/v1/rerank \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tomaarsen/Qwen3-Reranker-0.6B-seq-cls",
    "query": "什么是机器学习？",
    "documents": [ "机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式", "机器学习是一种编程语言，用于开发网站", "机器学习是数据库管理系统的一种", "机器学习是操作系统的一种类型" ]
  }'

vllm serve tomaarsen/Qwen3-Reranker-0.6B-seq-cls 

{"id":"rerank-4f53115e17674498a3a7317fa664ee11","model":"tomaarsen/Qwen3-Reranker-0.6B-seq-cls","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一
  类型"},"relevance_score":0.984752357006073},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9655764698982239},{"index":1,"document":{"text":"机器学习
  一种编程语言，用于开发网站"},"relevance_score":0.9000037312507629},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.783326268196106}]}


vllm serve tomaarsen/Qwen3-Reranker-0.6B-seq-cls --dtype float32


{"id":"rerank-68484c2c25494d67810d4ab6e49f26b2","model":"tomaarsen/Qwen3-Reranker-0.6B-seq-cls","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一
  类型"},"relevance_score":0.9847448468208313},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9657185673713684},{"index":1,"document":{"text":"机器学习
  一种编程语言，用于开发网站"},"relevance_score":0.9001880884170532},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.7831804156303406}]}




vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'



  curl http://127.0.0.1:8000/v1/rerank \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-Reranker-0.6B",
    "query": "什么是机器学习？",
    "documents": [ "机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式", "机器学习是一种编程语言，用于开发网站", "机器学习是数据库管理系统的一种", "机器学习是操作系统的一种类型" ]
  }'


vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'


  {"id":"rerank-44ace860b82f4d61aef0eedf4e4a763c","model":"Qwen/Qwen3-Reranker-0.6B","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一种类型"},"relevance_score":0.9840936064720154},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9648551344871521},{"index":1,"document":{"text":"机器学习是一种编程语
  ，用于开发网站"},"relevance_score":0.9005298018455505},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.7853085398674011}]}

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}' --dtype float32


{"id":"rerank-bbf8be7557734ff0904430b39a63fe4e","model":"Qwen/Qwen3-Reranker-0.6B","usage":{"total_tokens":53},"results":[{"index":3,"document":{"text":"机器学习是操作系统的一种类型"},"relevance_score":0.9847448468208313},{"index":2,"document":{"text":"机器学习是数据库管理系统的一种"},"relevance_score":0.9657185673713684},{"index":1,"document":{"text":"机器学习是一种编程语
  ，用于开发网站"},"relevance_score":0.9001880884170532},{"index":0,"document":{"text":"机器学习是人工智能的一个分支，通过算法让计算机从数据中学习模式"},"relevance_score":0.7831804156303406}]}

@noooop
也就是说我即便是在curl的测试过程中我也应该像这里一样对我的query和documents进行拼装，使用template组装我的问题和答案是吗？
prefix = '<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
suffix = "<|im_end|>\n<|im_start|>assistant\n\n\n\n\n"

query_template = "{prefix}: {instruction}\n: {query}\n"
document_template = ": {doc}{suffix}"

我想问一下这是Qwen3的rerank模型的特性还是什么原因呢，因为前面我在使用bge-rerank等其他rerank模型的时候，没有遇到该种情况

[noooop]

也就是说我即便是在curl的测试过程中我也应该像这里一样对我的query和documents进行拼装，使用template组装我的问题和答案是吗？

对

Qwen3-Reranker 是一种 llm as Reranker，可以理解为就是一个大语音模型，提示词就是让它输出 "yes" or "no". 没有这个提示词，它就不输出"yes" or "no" 了

具体请参考relevance generation

[MrBlue-1996]

万分感谢，我先处理一下 后面没问题的话关掉这个Issue 感谢！