Fast model loading

[lionelvillard]

I'd like to start the discussion on adding fast model loading capability to llm-d. model service seems the right starting point.

The goal is to ensure that safetensors model files are close to (ie. on the worker nodes) where vllm instances are created. When worker nodes have NVMe local storage, these files should be stored on it in order to enable direct transfer from storage to GPU.

GIE defines the concept of InferenceModel and whatever solutions we come up with should play nicely with this concept.

@sriumcp what's your opinion?

/cc @fabolive @manoelmarques @aavarghese

[sriumcp]

Yes, model caching (cluster and node level) are both in-scope and part of the roadmap.

Lets evolve a design that plays well with rest of llm-d.

[mgoin]

vLLM actually has a few different extensions for model loaders. One of them is the Run:ai Model Streamer. It uses multiple threads to read tensors concurrently from a file in some file or object storage to a dedicated buffer in the CPU memory.

[lionelvillard]

A lesser known model loader extension is fastsafetensor that can be used when NVMes are available.

[lionelvillard]

/cc @jimcadden @paulcastro

[jeremyeder]

@chcost mind pinging the coreweave team to work with us on this?

[wbrown]

@lionelvillard

@chcost mind pinging the coreweave team to work with us on this?

My recommendation would be https://github.com/coreweave/tensorizer

We just merged into vllm v1 main an update to make tensorizer work again with the new version of vLLM. It's integral to vllm.

[wbrown]

PR that was merged in: vllm-project/vllm#17926