[Bug]: `jinaai/jina-reranker-v2-base-multilingual` doesn't support long context above 1024 tokens

[distributedlock]

Your current environment

vLLM Production Stack on Kube with Helm.

Helm values:

    servingEngineSpec:
      runtimeClassName: ''
      modelSpec:
        - name: bge-reranker-v2-m3
          repository: vllm/vllm-openai
          tag: v0.9.1
          modelURL: BAAI/bge-reranker-v2-m3
          replicaCount: 1
          requestCPU: 8
          requestMemory: 16Gi
          requestGPU: 1
        - name: jina-reranker-v2-base-multilingual
          repository: vllm/vllm-openai
          tag: v0.9.1
          modelURL: jinaai/jina-reranker-v2-base-multilingual
          replicaCount: 1
          requestCPU: 8
          requestMemory: 8Gi
          requestGPU: 1
          vllmConfig:
            extraArgs:
              - --trust-remote-code

🐛 Describe the bug

jinaai/jina-reranker-v2-base-multilingual is no supporting long context reranking above 1024 with vLLM while outside vLLM (with Transformers) it will support it.

This is the error we get from /v1/rerank when invoking the request. We tested with BAAI/bge-reranker-v2-m3 which ends up working nicely without issue with long context (>1024 tokens), it's just long context mode (>1024 tokens) on the jina reranking model that fails.

BadRequestError: status_code: 400, body: {'object': 'error', 'message': "This model's maximum context length is 1024 tokens. However, you requested 1301 tokens in the input for embedding generation. Please reduce the length of the input.", 'type': 'BadRequestError', 'param': None, 'code': 400}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

"max_position_embeddings": 1026, in config.json.

https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual/blob/main/config.json

It couldn’t support a longer context.

[distributedlock]

Is this incorrect in their model README page?

The jina-reranker-v2-base-multilingual model is capable of handling long texts with a context length of up to 1024 tokens, enabling the processing of extensive inputs. To enable the model to handle long texts that exceed 1024 tokens, the model uses a sliding window approach to chunk the input text into smaller pieces and rerank each chunk separately.

Their model page https://jina.ai/models/jina-reranker-v2-base-multilingual/ also says it support long context.

[noooop]

README page

The jina-reranker-v2-base-multilingual model is capable of handling long texts with a context length of up to 1024 tokens, enabling the processing of extensive inputs.

A year ago, most models were 512, which might be considered relatively long.

[distributedlock]

https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual#:~:text=To%20enable%20the%20model%20to%20handle%20long%20texts%20that%20exceed%201024%20tokens%2C%20the%20model%20uses%20a%20sliding%20window%20approach%20to%20chunk%20the%20input%20text%20into%20smaller%20pieces%20and%20rerank%20each%20chunk%20separately.

Is this part not true? I think that's what the trust_remote_code=True is doing to enable the long context over 1024 tokens.

[noooop]

The rerank() function will automatically chunk the input documents into smaller pieces if they exceed the model's maximum input length. This allows you to rerank long documents without running into memory issues. Specifically, the rerank() function will split the documents into chunks of size max_length and rerank each chunk separately. The scores from all the chunks are then combined to produce the final reranking results. You can control the query length and document length in each chunk by setting the max_query_length and max_length parameters. The rerank() function also supports the overlap parameter (default is 80) which determines how much overlap there is between adjacent chunks. This can be useful when reranking long documents to ensure that the model has enough context to make accurate predictions.

You can implement it on the client side, but I think it won't yield good results.

[noooop]

try Qwen3 Reranker #19260

The previous model really performed poorly for long contexts (>512).

[distributedlock]

Thanks very much. We'll try that out. Your prompt response is much appreciated.