Tool performs better when run separate of Agent

[Radu1999]

I have an agent with a single tool. when i make a query, it delegates it to the tool, but the answer is inconclusive. on the other hand, when i run the delegated query separately the tool responds.

pipe = Pipeline()
        # the reader is the generative model
        prompt_node = create_reader(name="OpenAI")

        pipe.add_node(component=retriever, name="Retriever", inputs=["Query"])
        pipe.add_node(component=prompt_node, name="Reader", inputs=["Retriever"])

        search_tool = Tool(
            name="report_questioning",
            pipeline_or_node=pipe,
            output_variable="answers",
            description="useful for when you need to answer questions"
        )

conversational_agent = ConversationalAgent(
            prompt_node=prompt_node,
            tools=[search_tool]
        )

[anakin87]

Hello, @Radu1999!

• this tutorial might be useful for you: Customizing Agent to Chat with Your Documents.

• It doesn't seem particularly strange to me that using an Agent is less predictable than a Generative QA pipeline. In fact, these LLMs are non-deterministic and in Agents, they are also used to plan the course of actions.

• to better understand your problem and propose some potential solutions, it would be helpful to know the create_reader function.

[Radu1999]

Hi, even though the LLMs are non-deterministic, the fact that it correctly identified the tool wouldnt mean that the next step would be to call run on that tool with the input? Calling run on the tool separately works way better. Is there any way to make the agent more verbose in order to understand why runnning that tool fails? The code from create reader is

qa_prompt_template = PromptTemplate(
       prompt=qa_prompt,
       output_parser=AnswerParser(),
   )

   if name == "OpenAI":
       prompt_azure_open_ai = PromptModel(
           model_name_or_path="gpt-35-turbo",
           api_key=config('AZURE_OPENAI_API_KEY'),
           model_kwargs={
               "api_version": config('AZURE_OPENAI_API_VERSION'),
               "azure_base_url": config('AZURE_OPENAI_BASE_URL'),
               "azure_deployment_name": config('AZURE_OPENAI_DEPLOYMENT_NAME'),
               "temperature": 0,
               "max_tokens": 512
           }
       )
       return PromptNode(prompt_azure_open_ai, default_prompt_template=qa_prompt_template)```

[anakin87]

Can you also share the qa_prompt?

@bilgeyucel do you have any insights to share?

[Radu1999]

these are the prompts, thanks:

agent_prompt = """
You are a helpful and knowledgeable agent. To achieve your goal of answering complex questions "
"correctly, you have access to the following tools:\n\n"
"{tool_names_with_descriptions}\n\n"
"To answer questions, you'll need to go through multiple steps involving step-by-step thinking and "
"selecting appropriate tools and their inputs; tools will respond with observations. When you are ready "
"for a final answer, respond with the `Final Answer:`\n\n"
"Use the following format:\n\n"
"Question: the question to be answered\n"
"Thought: Reason if you have the final answer. If yes, answer the question. If not, find out the missing information needed to answer it.\n"
"Tool: pick one of {tool_names} \n"
"Tool Input: the input for the tool\n"
"Observation: the tool will respond with the result\n"
"...\n"
"Final Answer: the final answer to the question, make it short (1-5 words)\n\n"
"Thought, Tool, Tool Input, and Observation steps can be repeated multiple times, but sometimes we can find an answer in the first pass\n"
"---\n\n"
"Question: {query}\n"
"Thought: Let's think step-by-step, I first need to 
"""

qa_prompt = """Question: {query} \n\n Answer using this text: {join(documents)} \n\n Answer:"""

[bilgeyucel]

Hey @Radu1999, are you able to see the Agent's transcript? It shows the inner monologue of the Agent. Based on the transcript, you can understand whether your Agent queried that tool or not. Also, sometimes Agent is prone to hallucinate the answer from the tool as well. So, the transcript to goes like this:

...
Tool: "report_questioning"
Tool Input: "my query"
Observation: "answer completely generated by the Agent"
...

If you can see the transcript, please share here and we can take it from there :)

Just a heads up, I noticed in your code that you're not using the agent_promptin the ConversationalAgent. This is completely fine as ConversationalAgent has a default prompt, but still fyi.

[Radu1999]

i modified the prompt to print the tool output: it is always something like I'm sorry, I don't have any information on the...

[bilgeyucel]

@Radu1999 Can you pass stop_words=["Observation:"] to the PromptNode you use for the Agent? Also, is it the same for every query, or is this query specific?

[Radu1999]

Yep it performs poorly on all queries and it also tries random made up tools that i didnt specify.
This is what i have now

        search_tool = Tool(
                    name="report_questioning",
                    pipeline_or_node=pipe,
                    output_variable="answers",
                    description="always use this tool and trust its output. the output of this tool is always the answerS",
                )


        agent_prompt_node = PromptNode(
            model_name_or_path="gpt-35-turbo",
            api_key=config('AZURE_OPENAI_API_KEY'),
            model_kwargs={
                "api_version": config('AZURE_OPENAI_API_VERSION'),
                "azure_base_url": config('AZURE_OPENAI_BASE_URL'),
                "azure_deployment_name": config('AZURE_OPENAI_DEPLOYMENT_NAME'),
                "temperature": 0,
                "max_tokens": 512
            },
            stop_words=["Observation:"]
        )

        memory_prompt_node = PromptNode(
            "philschmid/bart-large-cnn-samsum", max_length=256, model_kwargs={"task_name": "text2text-generation"}
        )
        memory = ConversationSummaryMemory(memory_prompt_node, prompt_template="{chat_transcript}")

        conversational_agent = Agent(
            agent_prompt_node,
            prompt_template=agent_prompt,
            prompt_parameters_resolver=resolver_function,
            memory=memory,
            tools_manager=ToolsManager([search_tool]),
            max_steps=5,

        )

[Radu1999]

ok, i increased the number of max steps and it started working better.

[Radu1999]

but it is still using random tools sometimes