Agent capable of filling forms

[gianfrancodemarco]

Hello,

I'm trying to build a set of tools that are more complex that common tools.
In particular, these can be seen as forms that the user has to fill before the tool can be used, so the agent should be able to proactively ask the user for the needed information.
At the moment i'm going with exending the AgentExecutor and i need to update the LLMChain at runtime.

My question is, is there anything already in Langchain to support this kind of task (form filling) that I could use as guidance?

Thanks

[dosubot[bot]]

🤖

To better assist you with your query, could you please provide more details about the specific functionalities you're trying to implement? For instance, what kind of information are you looking to collect from the users through these forms? Also, could you elaborate on what you mean by "updating the LLMChain at runtime"? Any additional context or examples would be greatly appreciated. This will help me understand your requirements more clearly and provide a more accurate response.

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[gianfrancodemarco]

@dosu-bot I would like to implement an agent that has:

• normal tools
• form tools

Form tools are tools that require more complex params to be called, possibly with validation: e.g. name, surname, address, phone etc.
The agent should be able to understand that the user wants to use these tools, and start asking for relevant information until it has it all.

Also, could you elaborate on what you mean by "updating the LLMChain at runtime"?
At the moment, my implementation involves updating the LLMChain from inside the AgentExecutor to change the prompt at runtime, which is not really elegant.

[dosubot[bot]]

🤖

Yes, LangChain does provide a built-in feature for implementing an agent that can understand when a user wants to use complex tools requiring additional parameters. This is done through the create_structured_chat_agent function. This function creates an agent aimed at supporting tools with multiple inputs.

The create_structured_chat_agent function takes three parameters: llm, tools, and prompt. The llm parameter is the language model to use as the agent. The tools parameter is the tools this agent has access to. The prompt parameter is the prompt to use, which must have input keys of tools, tool_names, and agent_scratchpad.

Here is the relevant code:

def create_structured_chat_agent(
    llm: BaseLanguageModel, tools: Sequence[BaseTool], prompt: ChatPromptTemplate
) -> Runnable:
    """Create an agent aimed at supporting tools with multiple inputs.

    Args:
        llm: LLM to use as the agent.
        tools: Tools this agent has access to.
        prompt: The prompt to use, must have input keys of
            `tools`, `tool_names`, and `agent_scratchpad`.

    Returns:
        A runnable sequence representing an agent. It takes as input all the same input
        variables as the prompt passed in does. It returns as output either an
        AgentAction or AgentFinish.

    """
    missing_vars = {"tools", "tool_names", "agent_scratchpad"}.difference(
        prompt.input_variables
    )
    if missing_vars:
        raise ValueError(f"Prompt missing required variables: {missing_vars}")
    prompt = prompt.partial(
        tools=render_text_description_and_args(list(tools)),
        tool_names=", ".join([t.name for t in tools]),
    )
    llm_with_stop = llm.bind(stop=["Observation"])

    agent = (
        RunnablePassthrough.assign(
            agent_scratchpad=lambda x: format_log_to_str(x["intermediate_steps"]),
        )
        | prompt
        | llm_with_stop
        | JSONAgentOutputParser()
    )
    return agent

This function first checks if the prompt has the required input variables. If not, it raises a ValueError. Then it partially applies the prompt with the tools and tool names. It binds the stop condition to the language model and finally constructs the agent by chaining the RunnablePassthrough, prompt, language model with stop condition, and JSONAgentOutputParser. The resulting agent can handle complex tools with multiple inputs.

You can find more details in the LangChain repository.

The create_structured_chat_agent function in LangChain handles the collection of additional parameters required by complex tools by using the prompt parameter. This function expects a ChatPromptTemplate object as the prompt parameter.

The ChatPromptTemplate object must have input keys of tools, tool_names, and agent_scratchpad. The function checks if these keys are present in the prompt object's input_variables attribute. If any of these keys are missing, the function raises a ValueError.

The tools key is used to pass the tools' descriptions and arguments, which are rendered as text using the render_text_description_and_args function. The tool_names key is used to pass the names of the tools, which are joined into a string separated by commas. The agent_scratchpad key is used to pass the agent's scratchpad, which is formatted as a string using the format_log_to_str function.

This approach should provide a more elegant solution to your problem than extending the AgentExecutor and updating the LLMChain at runtime.

Sources

• libs/langchain/langchain/agents/structured_chat/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[gianfrancodemarco]

I made good progress in making this work.
Basically, i built this graph with LangGraph:

Users can define FormTools which have 3 states:

When a FormTool is active, the other (inactive) FormTools are hidden in order to save time/computation and improve performances.
The FormAgent is an agent with specific prompts that ask the user to fill in the information, one by one. When all the information is collected, it asks the user to confirm it before actually executing the action.

If any of this seems interesting I can try to share the code

[BrunoRoth95]

Hey @gianfrancodemarco

Congrats for this amazing project! Form filling is a thing I was trying to do for some time and it seems there's not much references about it.

I'm trying to adapt your notebook to my use case but it's been hard because of the complexity of the code. Do you have any new updates or improvement of this notebook?

[gianfrancodemarco]

Hi @BrunoRoth95, I have a repo with the code organized better (mainly, it's not all in one file), but I think it's private and idk if I can still make it public. I'll check in the next days.
However if you want, I can try to give some pointings for what you're trying to di

[LatVAlY]

Hey @gianfrancodemarco
Congrats, it looks an awesome project! I'm very interested on the code.

[xiyu-205]

Same here, @gianfrancodemarco the project is amazing and very inspiring.

It's just little out-dated because Langchain got updated, which some packages are deprecated - namely ToolExecutor, ToolInvocationInterface in langgraph.prebuilt.

Interested in the repo you mentioned above, if by any chance you could share.

Thanks a ton

[gianfrancodemarco]

Hi @LatVAlY and @xiyu-205, thanks for your interest!
I'll share the repos since, as you've already noted, my code is probably really outdated now - I think LangChain has released more libs and architectures to do what I was doing from scratch
I have a lib with the core logic here: converso
And a telegram bot using it: converso-chatbot

[aritrakar]

Hey @gianfrancodemarco , thanks for your help, it worked! For my use case, I need dynamic forms (fields defined in a YAML template file) which I was able to get working. Essentially, I want to create a general form-filling tool to make available to users for any kind of form that can be defined via YAML template files. It works, but I'm having trouble with the following:

1. Required vs optional fields: I spent a lot of time playing around with Pydantic's Field class, but to no end. The closest I came to implementing required fields was to modify how the "filled" state was being determined in your code, i.e. checking if all fields had some value (which leads me to the second point below), but wasn't able to get a working version
2. I also want to be able to specify default values for both required and optional fields so that the user can say something like "use the default value for this field" (for cases where they don't quite know what value to input). But setting a field as "required" prevents me from doing so.

I would be grateful if you could provide any pointers on these. Thank you.

[gianfrancodemarco]

How do you expect optional fields to work? They are not asked to the user but if they explicitly tell to assign a value it should work.
If this is the case, I guess you can omit them from the get_next_field_to_collect and assign the default value directly in the code.
If explicitly told, the LLM should be able to update their value still.

[LuisFX]

@gianfrancodemarco This is fantastic! I'm also in the process of adding a chatbot that automatically fills fields in forms for site feedback, profile details, etc... I haven't gotten the project to run yet, but I would definitely be interested in collaborating on this project. I'll try to get it running tonight.

Cheers

ps: At first glance, I'm assuming the starting point is the converso-chatbot repo. Using the converso api, and the converso_telegram as the client?

pss: If it's not too much to ask, it'd be great if there was details to get it all running locally on the readme. Greatly appreciated!

[gianfrancodemarco]

Hi @LuisFX,
I've answered on the issue you opened.
Btw you are correct. The converse repo is the core package; the converso-chatbot is a whole project with the infrastructure that leverages the converso package to run a telegram bot.