How to start inference serving through LLM object

[Jiayi-Pan]

I'm working on a project that involves performing multi-turn model evaluations during model training.

The majority of these evaluation scripts are all based on the OpenAI API format. It would be really helpful if I could start and stop the vllm OpenAI-compatible inference server programmatically, instead of using the command line.

I am new to vllm and am curious if there is a built-in way to achieve this. If not, how challenging would it be to implement this functionality ourselves?

Here is a code example illustrating what I am aiming for:

llm = LLM(model=model)
llm.serve(port_num, **serve_args)  # Start serving
results = []

# In reality, we would call complex_eval in parallel
for i in range(num_evals):
    results.append(complex_eval(i, port_num))

llm.stop()  # Stop serving

Any guidance or suggestions would be greatly appreciated!

[Jiayi-Pan]

actually vllm project allows discussing usage-related questions in github issues. I've moved the question there #5227