Support for Custom trained llm models from hugging face

[shrirudragoud]

I am Training my own model using the hugging face mistral llm and i want to know how can i use the vllm for my own trained model which i can run on my own onprem server

[chenxu2048]

If the architecture of your model remains unchanged during training, it is supported in vLLM. You can check out all the supported models at this page.

[mcpark84]

I've encountered the same issue. Generally, when we fine-tune a model based on what Hugging Face provides and save it, a file in a format that can be loaded by Hugging Face is generated. How can I use such a saved model in vLLM?