How do you implement Continuous batching of incoming requests?

[wqh17101]

Where can i find the codes about it? I am really interested about it.

[irasin]

mark

[lucasjinreal]

Looks like doesn't support batch, for example

def forward(
        self,
        hidden_states: torch.Tensor,
        positions: torch.LongTensor,
        kv_cache: KVCache,
        input_metadata: InputMetadata,
        cache_event: Optional[torch.cuda.Event],
    ) -> torch.Tensor:

the Attention hiddenstate here has no batch dim

[zhuohan123]

Please check the logic starting at this step function:

vllm/vllm/engine/llm_engine.py

Line 228 in b4b195b

def step(self) -> List[RequestOutput]:

Move this issue to discussions for future questions