How to load a fine-tuned model to vllm from the supported archs?

[brando90]

I've fine-tuned say mistral & gemma 7B and wish to use those weights in vllm. Is that possible to be done? How does one do this?

Thanks in advance! :)

[ywang96]

Have you tried LLM(model=<path to the directory with your weights>, ...)? If your model is fine-tuned from one of the those supported models and the directory has all the necessary files (weights, config files, etc), then this should work.

[brando90]

@ywang96 can one pass directly the hugging face HF model object to vllm?

e.g.,

from transformers import AutoModelForCausalLM, AutoTokenizer
# Assuming VLLM can be initialized with a transformers model object directly. 
# If not, this approach needs adjustment based on VLLM's API.

def download_model_and_tokenizer(model_name: str):
    """
    Downloads a fine-tuned model and its tokenizer from Hugging Face.
    
    Args:
        model_name (str): The name of the model on Hugging Face.
    
    Returns:
        The downloaded model and tokenizer.
    """
    model = AutoModelForCausalLM.from_pretrained(model_name)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    return model, tokenizer

model_name = 'morph-labs/morph-prover-v0-7b'
model, tokenizer = download_model_and_tokenizer(model_name)

# Here, adjust according to how VLLM expects to load the model. If VLLM cannot directly accept a model object,
# and requires a file path instead, this part needs to be aligned with VLLM's loading mechanism.
# For demonstration purposes, the below code assumes a hypothetical scenario.
# from vllm import VLLM
# llm = VLLM(model=model)  # Hypothetical example if VLLM can accept a model object directly.
# print("Model loaded successfully into VLLM.")

if not, do you have a demonstration of how to do your suggestion?

[ywang96]

If your model is already on huggingface, you should be able to pass the model id to LLM like LLM(model=morph-labs/morph-prover-v0-7b, ...)

[brando90]

@ywang96 sort of worked? But output is weird:

def _test_morph_vllm_ouput():
    try:
        base_model_name='mistralai/Mistral-7B-Instruct-v0.1'
        base_model_name='mistralai/Mistral-7B-Instruct-v0.1'
        # merged_peft_model_name="morph-labs/morph-prover-v0-7b"
        merged_peft_model_name = base_model_name
        llm = LLM(model=merged_peft_model_name, tokenizer=base_model_name)
        outputs = llm.generate(["Are you Morph Prover? "])

        prompt = outputs[0].prompt
        generated_text = outputs[0].outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
        print("Model loaded into VLLM successfully.")
    except FileNotFoundError as e:
        print(str(e))

if __name__ == "__main__":
    import time
    start_time = time.time()
    main()
    end_time = time.time()
    print(f"Execution Time: {end_time - start_time:.2f} sec")

output:

INFO 03-29 23:45:44 model_runner.py:756] Graph capturing finished in 4 secs.
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.00it/s]
Prompt: 'Are you Morph Prover? ', Generated text: "🤖\n===========================\n\nHello, I'm Mor"
Model loaded into VLLM successfully.
Execution Time: 13.77 sec

[brando90]

ref: https://medium.com/@crismunozv/using-fine-tuned-llm-with-vllm-ee34e7db5495

[ywang96]

I recommend specify sampling_params when you call .generate rather than using default values (e.g, default temperature is 1.0) - you may want to take a look at it here.