You should consider increasing max_length or, better yet, setting max_new_tokens.

[reemas-irasna]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

base_model = "/kaggle/input/llama-3.1/transformers/8b-instruct/1"
# QLoRA config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch_dtype,
    bnb_4bit_use_double_quant=True,
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="cuda:0",
    attn_implementation=attn_implementation
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

pipe = pipeline(
        task="text-generation", 
        model=model, 
        tokenizer=tokenizer, 
        do_sample=True, 
        top_k=1,
        temperature=0.5,
        top_p=0.95,
        num_return_sequences=1,
        eos_token_id=tokenizer.eos_token_id,
        max_new_tokens=2048,
    )
from langchain_community.llms import HuggingFacePipeline
# Create a Hugging Face pipeline for local language model (LLM) using the 'pipe' pipeline
local_llm = HuggingFacePipeline(pipeline=pipe, pipeline_kwargs={"max_new_tokens": 6144})

from langchain.agents.agent_types import AgentType
from langchain_experimental.agents import create_pandas_dataframe_agent
from langchain_experimental.tools import PythonREPLTool
import pandas as pd 
df = pd.read_csv('/kaggle/input/can-data/decoded_data.csv') 

tools = [PythonREPLTool()]

eda_agent = create_pandas_dataframe_agent(local_llm, df[:10],
                                # tools = tools,
                                verbose=True,
                                allow_dangerous_code=True,
                                agent_executor_kwargs={"handle_parsing_errors": True, "max_new_tokens": 6144, "max_length": 6144},
                                agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION)

insight = """Analyze the data to identify possible thermal imbalances\n
            or overheating events in the battery system."""


result = eda_agent.invoke(insight)

Description

I am using one of langchain agent i.e. create_pandas_dataframe_agent, loading llama3.1 8b instruct model with 4 bit and bytes configuration. While using the agent im getting error to increase max_length or max_new_tokens. Even though i am providing it in pipeline, agent defination, and as we know llama 3 have 8k limit but still it is not going correctly, please check the code of mine correct me i am lagging somewhere

complete error-

Setting pad_token_id to eos_token_id:None for open-end generation.
/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py:1220: UserWarning: Using the model-agnostic default max_length (=20) to control the generation length. We recommend setting max_new_tokens to control the maximum length of the generation.
warnings.warn(
Exception in thread Thread-7 (generate):
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/opt/conda/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/opt/conda/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1906, in generate
self._validate_generated_length(generation_config, input_ids_length, has_default_max_length)
File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1228, in _validate_generated_length
raise ValueError(
ValueError: Input length of input_ids is 4117, but max_length is set to 4096. This can lead to unexpected behavior. You should consider increasing max_length or, better yet, setting max_new_tokens.

System Info

Python 3.10.14
langchain Version: 0.3.1
langchain-experimental Version: 0.3.2
torch - 2.4.1+cu121

[dosubot[bot]]

Hey there, @reemas-irasna! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to solve your issue.

I found a similar closed issue that might be relevant to your problem: Chat with pandas df string length BadRequestError, which was last updated on September 03, 2024 ^[1].

To address the error you're encountering, ensure that the max_new_tokens parameter is set correctly in your code. The ExLlamaV2 class in the LangChain framework has a max_new_tokens field, which is set to a default value of 150. You can adjust this value according to your needs. Here's an example of how you might set the max_new_tokens parameter when initializing the model:

from langchain_community.llms import Exllamav2

llm = Exllamav2(model_path="/path/to/llama/model", max_new_tokens=200)

In your case, you have already set max_new_tokens to 6144 in multiple places, which should be sufficient. However, the error message indicates that the max_length is set to 4096, which is less than the input length of 4117. You should ensure that the max_length parameter is also set appropriately to accommodate the input length. You can try setting max_length to a higher value than the input length to avoid this error.

To continue talking to Dosu, mention @dosu.

[reemas-irasna]

can i use this with create_pandas_dataframe_agent?