[RFC][Feature] Support Remote Prefill in PD Disaggregation

[rainj-me]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

With middle size(30B-70B) LLM, not all requests benefit from PD disaggregation (e.g. input tokens <= 128, output tokens >=512). In this case, remote prefill with conditional disaggregation is a nice to have feature.

Related resources

Dynamo implement remote prefill in vLLM v0/v1

• https://github.com/ai-dynamo/dynamo/blob/main/docs/architecture/disagg_serving.md
• [P/D] NIXL Integration vllm-project/vllm#17751
• [NIXL] vllm v0 nixl integration vllm-project/vllm#16677

[rainj-me]

Remote Prefill workflow

• the orange dash flow only exist on inference engine startup
• the green dash flow only exist on when the decode agent information is not existing in the Prefill TP rank
• the blue flow is the KvCache transfer and the first token id transfer