Regarding tutorial '25_Customizing_Agent'

[dkbs12]

Hello,
When I study tutorial '25_Customizing_Agent', I face some difficulties as below.
Originally they use the model 'text-davinci-003' of OpenAI in this tutorial, but I replace it to 'google/flan-t5-large' in PromptNode.
And I found a warning as "Token indices sequence length is longer than the specified maximum sequence length for this model (560 > 512). Running this sequence through the model will result in indexing errors"
My first question is how to prevent such a warning message and solve this maximum length exceeding problem.
(The source of dataset is "bilgeyucel/seven-wonders")

The second question is how to make it available using 'google/flan-t5-large' model to answer properly to the question which need an answer not in the documents.
In this tutorial, I used the question "How does Taylor Swift look like?" and I expected the answer as 'Answering is not possible given the available information.', but the generated answer was "The Great Pyramid was raised to prevent the lower classes from remaining unoccupied."
I write down the coding and the results as below for your reference;

< Coding >
from haystack.nodes import PromptNode, PromptTemplate, AnswerParser, BM25Retriever
from haystack.pipelines import Pipeline

retriever = BM25Retriever(document_store=document_store, top_k=2)

prompt_template = PromptTemplate(
prompt="""Synthesize a comprehensive answer from the following text for the given question.
Provide a clear and concise response that summarizes the key points and information presented in the text.
Your answer should be in your own words and be no longer than 50 words.
\n\n Related text: {join(documents)} \n\n Question: {query} \n\n Answer:""",
output_parser=AnswerParser(),
)

prompt_node = PromptNode(
model_name_or_path="google/flan-t5-large", default_prompt_template=prompt_template
)

generative_pipeline = Pipeline()
generative_pipeline.add_node(component=retriever, name="retriever", inputs=["Query"])
generative_pipeline.add_node(component=prompt_node, name="prompt_node", inputs=["retriever"])

from haystack.utils import print_answers

response = generative_pipeline.run("How does Rhodes Statue look like?")
print_answers(response, details="minimum")

< Results >
WARNING:haystack.nodes.prompt.invocation_layer.hugging_face:The prompt has been truncated from 560 tokens to 412 tokens so that the prompt length and answer length (100 tokens) fit within the max token limit (512 tokens). Shorten the prompt to prevent it from being cut off
'Query: How does Rhodes Statue look like?'
'Answers:'
[ { 'answer': 'The head would have had curly hair with evenly spaced '
'spikes of bronze or silver flame radiating, similar to the '
'images found on contemporary Rhodian coins.'}]

< Coding >
response = generative_pipeline.run("How does Taylor Swift look like?")
print_answers(response, details="minimum")

< Results >
WARNING:haystack.nodes.prompt.invocation_layer.hugging_face:The prompt has been truncated from 602 tokens to 412 tokens so that the prompt length and answer length (100 tokens) fit within the max token limit (512 tokens). Shorten the prompt to prevent it from being cut off
'Query: How does Taylor Swift look like?'
'Answers:'
[ { 'answer': 'The Great Pyramid was raised to prevent the lower classes '
'from remaining unoccupied.'}]

Thanks.

[vblagoje]

Unfortunately, @dkbs12 none of these models are good in reasoning to run Agents. For now, these have to be OpenAI models. I guess that Claude and Cohere Command should work as well. Llama 2 has the highest chance of running agents successfully, and I have seen people online claim that, but we yet have to try it out.

[dkbs12]

Hi, vblagoje.
Thank you for your quick reply and your kind and professional explanation.
I'm a little bit disappointed, but I expect they may solve this problem in the near future.
By the way, do you have any idea how can I solve maximum length exceeding problem which pop up in the result as warning message which is "Token indices sequence length is longer than the specified maximum sequence length for this model (560 > 512). Running this sequence through the model will result in indexing errors"? (It was my first question that I wrote above)

In addition, I have another question about the 'answers' which comes from the result after running generative_pipeline.
I found 'None' in the items of score, context, offsets_in_document and offsets_in_context as below.
Why is all of them 'None'?

< Result >

{'answers': [<Answer {'answer': 'The head would have been a standard rendering at the time of the Arab wars.', 'type': 'generative', 'score': None, 'context': None, 'offsets_in_document': None, 'offsets_in_context': None,

Thank you for your help!