Langchain with Local LLMs as API

[RamishSiddiqui]

Has anybody tried to work with langchains that call locally deployed LLMs on my own machine. Basically langchain makes an API call to Locally deployed LLM just as it makes api call with OpenAI ChatGPT but in this call the API is local.
Any help in this regard, like what framework is used to deploy LLMs as API and how langchain will call it ?

[hinthornw]

You can set an openai_proxy or configure the base_url when initializing

[minhbtrc]

You can use HuggingFacePipeline like this

from langchain import PromptTemplate, LLMChain, HuggingFacePipeline
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    eos_token_id = tokenizer.eos_token_id,
    device_map='auto',
    streamer=streamer,
    **generation_config
)
llm=HuggingFacePipeline(pipeline=pipe)

llm_chain = LLMChain(
    prompt=prompt,
    llm=llm,
    output_key="generated_text"
)

[RamishSiddiqui]

How will this work ? Suppose i have an LLM deployed at my localhost:12345 as an API.

[minhbtrc]

So you can add your model call to `_call' function in class CustomLLM

Example:

class CustomLLM(LLM):
    n: int

    def __init__(self):
        self.api_url = "XXXX"

    def make_request(self, prompt):
        # TODO: Implement api call to defployed model
        pass

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        return self.make_request(prompt)

[minhbtrc]

And them create your chain

your_model = CustomLLM(**kwargs) #<- add your params here
chain = LLMChain(
   prompt=prompt,
    model = your_model
)