"The prompt or the messages are too long"

[jambudipa]

Hello all 👋🏻

I'm getting this error running my prompt_node node:

Exception: Exception while running node 'prompt_node': The prompt or the messages are too long (1732 tokens). The length of the prompt or messages and the answer (32768 tokens) should be within the max token limit (32768 tokens). Reduce the length of the prompt or messages.

Struggling to see what these numbers are telling me. A little confusing?

Using gpt-4-32k like this:

prompt_node = PromptNode(
    "gpt-4-32k",
    default_prompt_template=lfqa_prompt,
    api_key=api_key,
    max_length=32768,
    model_kwargs={"stream": True, "stream_handler": custom_handler},
)

So,

• why does it complain at 1732 tokens, when the max is 32768?
• is there anything I can do to change this behaviour?

Having this issue with v1.8.0rc0, but it was the same with .16

[julian-risch]

Hi @jambudipa based on your question here, we updated the documentation: https://docs.haystack.deepset.ai/docs/prompt_node#usage
max_length only sets the maximum number of tokens for the generated text output. The token length of the prompt plus the token length of the generated text output together must not be larger than the overall number of tokens the LLM can process, which is 32768 tokens in your example. So you need to set max_length to 32768 minus the number of tokens in your prompt.
I agree that The prompt or the messages are too long (1732 tokens) is misleading/confusing.