Advancing Tool Support - Part 5 #2162
Conversation
* Introduced new ToolParam annotation for defining a description for tool parameters and marking them as (non)required. * Improved the JSON Schema generation for tools, solving inconsistencies between methods and functions, and ensuring a predictable outcome. * Added support for returning tool results directly to the user instead of passing them back to the model. Introduced new ToolExecutionResult API to propagate this information. * Consolidated naming of tool-related options in ToolCallingChatOptions. * Fixed varargs issue in ChatClient when passing ToolCallback[]. * Introduced new documentation for the tool calling capabilities in Spring AI, and deprecated the old one. * Bumped jsonschema dependency to 4.37.0. Relates to spring-projectsgh-2049 Signed-off-by: Thomas Vitale <ThomasVitale@users.noreply.github.com>
| var toolExecutionResult = this.toolCallingManager.executeToolCalls(prompt, response); | ||
| if (toolExecutionResult.returnDirect()) { | ||
| // Return tool execution result directly to the client. | ||
| return ChatResponse.builder() |
There was a problem hiding this comment.
When a given tool is marked with the option returnDirect, the tool call result is sent back to the user directly, wrapping each tool call result in a Generation object.
That is especially useful when building agents. For example, if I have a RAG tool, I might want to get the result back directly instead of having the main agent model post-process the RAG answer. Or I might want certain tools to end the reasoning loop of an agent.
| @@ -216,9 +216,9 @@ interface ChatClientRequestSpec { | |||
|
|
|||
| ChatClientRequestSpec tools(String... toolNames); | |||
|
|
|||
| ChatClientRequestSpec tools(Object... toolObjects); | |||
| ChatClientRequestSpec tools(FunctionCallback... toolCallbacks); | |||
There was a problem hiding this comment.
The Java compiler complaints when trying to pass an array of FunctionCallback and requires casting. In this way, we keep the same API, but it's more explicit and no casting is required.
| @@ -53,12 +53,12 @@ public interface ToolCallingChatOptions extends FunctionCallingOptions { | |||
| /** | |||
| * Names of the tools to register with the ChatModel. | |||
| */ | |||
| Set<String> getTools(); | |||
| Set<String> getToolNames(); | |||
There was a problem hiding this comment.
Having the same API (tools()) in ChatClient and ToolCallingChatOptions, but with very different behaviours, didn't seem good.
Now, the ChatClient keeps having one single API with overloaded methods to support different use cases: tools().
ToolCallingChatOptions, instead, reflects the two types of tool-related information it stores: toolNames() and toolCallbacks().
@tzolov what do you think?
| @@ -39,7 +38,7 @@ public interface ToolCallingManager { | |||
| /** | |||
| * Execute the tool calls requested by the model. | |||
| */ | |||
| List<Message> executeToolCalls(Prompt prompt, ChatResponse chatResponse); | |||
| ToolExecutionResult executeToolCalls(Prompt prompt, ChatResponse chatResponse); | |||
There was a problem hiding this comment.
I introduced this to propagate back whether the "returnDirect" option is enabled. Even without that use case, I think it's still a better return type than what I used before (List<Message>). It's more easily understandable.
| @Target({ ElementType.PARAMETER, ElementType.FIELD, ElementType.ANNOTATION_TYPE }) | ||
| @Retention(RetentionPolicy.RUNTIME) | ||
| @Documented | ||
| public @interface ToolParam { |
There was a problem hiding this comment.
For the naming, I followed the most used convention in Spring Framework (@RequestParam @CookieParam...).
I made it tool-specific, but this could actually be used also when using Structured Outputs (as well as the entire JSON schema generation logic in JsonSchemaGenerator). But if we want that, we need to come up with a better name).
| * to work well in the context of tool calling and structured outputs, aiming at ensuring | ||
| * consistency and robustness across different model providers. | ||
| * <p> | ||
| * Metadata such as descriptions and required properties can be specified using one of the |
There was a problem hiding this comment.
This was the main inconsistency I fixed in this PR. Methods and functions were treated differently, with functions having undocumented broader support for annotations. Now there's parity of behaviour between them when it comes to description and required properties.
|
Rebased and merged at 560baaa |
|
related to epic #2207 |
@ToolParamannotation for defining a description for tool parameters and marking them as (non)required.ToolExecutionResultAPI to propagate this information.ToolCallingChatOptions.ChatClientwhen passingToolCallback[].Relates to gh-2049