How do I use a custom LLama model as generator within the GenerativeQAPipeline?

[gooseillo]

Hi every,

I've started using Haystack recently for semantic search and its awesome!

Recently, I was able to load a Llama type model called Vicuna (quantized) and was able to load it locally. How do I use it in Haystack within the generator object?

generator = Seq2SeqGenerator(model_name_or_path=***use Vicuna model here**)
pipe = GenerativeQAPipeline(generator, retriever)

Any help will be appreatiated!

[bogdankostic]

Hi @gooseillo! The Seq2SeqGenerator is deprecated, we advise to use the PromptNode, which should support the vicuna model. Have a look at this tutorial on how to build a generative QA pipeline with PromptNode.

[gooseillo]

Hi @bogdankostic, I tried the turorial but got stuck at this step

from haystack.nodes import PromptNode, PromptTemplate

lfqa_prompt = PromptTemplate(
    name="lfqa",
    prompt_text="""Synthesize a comprehensive answer from the following text for the given question. 
                             Provide a clear and concise response that summarizes the key points and information presented in the text. 
                             Your answer should be in your own words and be no longer than 50 words. 
                             \n\n Related text: {join(documents)} \n\n Question: {query} \n\n Answer:""",
)

prompt_node = PromptNode(model_name_or_path="models\vicuna-13B-1.1-GPTQ-4bit-128g.GGML.bin")

This is the error I got -
ValueError: Model models�icuna-13B-1.1-GPTQ-4bit-128g.GGML.bin is not supported - no invocation layer found. Currently supported models are: [, ] Register a new invocation layer for models�icuna-13B-1.1-GPTQ-4bit-128g.GGML.bin using the register method.

I also tried it using the huggingface link

prompt_node = PromptNode(model_name_or_path="models\vicuna-13B-1.1-GPTQ-4bit-128g.GGML.bin", default_prompt_template=lfqa_prompt)

I'm able to run this model seperatly using the llama.cpp library.

[gooseillo]

I also attempted to install the latest version in colab using

!pip install git+https://github.com/deepset-ai/haystack.git@main

Then I ran this

from haystack.nodes import PromptNode

prompt_node = PromptNode(model_name_or_path="anon8231489123/vicuna-13b-GPTQ-4bit-128g")

and got this error-

ValueError: Could not load model anon8231489123/vicuna-13b-GPTQ-4bit-128g with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.auto.modeling_tf_auto.TFAutoModelForCausalLM'>, <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>).

[bogdankostic]

Sorry for the confusion @gooseillo! We don't support llama.cpp models in Haystack, we only support models that can be loaded using Hugging Face transformers. The second model you tried (anon8231489123/vicuna-13b-GPTQ-4bit-128g) can not be loaded with Hugging Face either.

One vicuna model that can be loaded using Hugging Face and is therefore supported by our PromptNode is for example this one: CarperAI/stable-vicuna-13b-delta.