Poor Performance on HF Pipeline

[bezir]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

HF Pipeline LangChain:

from langchain_huggingface.llms import HuggingFacePipeline
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "google/gemma-2-9b-it"

llm = HuggingFacePipeline.from_model_id(
    model_id=model_id,
    task="text-generation",
    device=0,
    pipeline_kwargs={"max_new_tokens": 100, "do_sample": True},
)

question = ... # Non-English question

llm(question)

HF Generation:

import torch
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="google/gemma-2-9b-it",
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="cuda",
    do_sample=True
)

messages = [
    {"role": "user", "content": question},
]

outputs = pipe(messages, max_new_tokens=100)
assistant_response = outputs[0]["generated_text"][-1]["content"].strip()
assistant_response

Description

I have unreliable and poor inference results when I use LangChain HF Pipeline. Both inference demos have the same parameters, although the second one is reliable and accurate whereas the first code produces unreliable and noisy content. I have tried multiple times/multiple prompts in the same setup.

System Info

I use the latest version of the LangChain in linux.

[dosubot[bot]]

Hey @bezir! I'm here to help you with any bugs, questions, or contributions you have for the LangChain repository. Let's tackle this issue together!

The LangChain HuggingFacePipeline might produce unreliable and noisy content compared to the direct HuggingFace pipeline due to differences in how the pipeline_kwargs are handled and passed to the underlying HuggingFace pipeline. The LangChain implementation has additional layers of abstraction and potential modifications to the parameters, which can introduce variability.

In the LangChain implementation, the pipeline_kwargs are managed and potentially overridden in multiple places, which might lead to inconsistencies. Additionally, the LangChain implementation processes responses in a batch and handles different tasks with specific logic for each, which might introduce noise or inconsistencies.

These additional layers and handling mechanisms can introduce variability and noise, making the output less reliable compared to using the HuggingFace pipeline directly with the same parameters and model ^[1][2].

To continue talking to Dosu, mention @dosu.