Using VLLM with Langchain for RAG purposes

[sadrafh]

Hello,

I am using VLLM to use Llama models for RAG purposes. However, I am constantly facing a runnable error. This is my VLLM model initialization:

llm_vllm = LLM(
model="Llama-2-7b-chat-hf",
device="cuda"
)
When I try to create a chain with:

chain = (
{"context": retriever, "question": RunnablePassthrough()} | prompt | llm_vllm | StrOutputParser()
)
response = chain.invoke(user_question)

I get the following error: TypeError: Expected a Runnable, callable or dict. Instead got an unsupported type: <class 'vllm.entrypoints.llm.LLM'>

Similarly, if I use:

from langchain.chains.question_answering import load_qa_chain
chain = load_qa_chain(llm_vllm, chain_type="stuff")
I get an error: llm instance of Runnable expected.

Is there any solution for that?

[mgoin]

Duplicate with issue here #5572