Add `builtin_tools` to `Agent` #1722

Kludex · 2025-05-14T09:00:44Z

No description provided.

Kludex · 2025-05-14T09:01:38Z

pydantic_ai_slim/pydantic_ai/agent.py

+        for tool in builtin_tools:
+            if tool == 'web-search':
+                self._builtin_tools.append(WebSearchTool())
+            else:
+                self._builtin_tools.append(tool)


It's easier to not have to handle string on the models, so we already do the transformation here.

Kludex · 2025-05-14T09:02:01Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+    """
+
+
+class UserLocation(TypedDict, total=False):


It's easier to handle this in the models if it's a TypedDict, since it matches the type.

Kludex · 2025-05-14T09:02:23Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+@dataclass
+class WebSearchTool(AbstractBuiltinTool):


From the DX POV, it's nicer for it to be a BaseModel or dataclass.

github-actions · 2025-05-14T09:03:07Z

Docs Preview

commit:	`800a71a`
Preview URL:	https://88c45a71-pydantic-ai-previews.pydantic.workers.dev

Kludex · 2025-05-14T09:03:26Z

pydantic_ai_slim/pydantic_ai/models/openai.py

        tools = list(model_settings.get('openai_builtin_tools', [])) + tools
+        tools = self._get_builtin_tools(model_request_parameters) + tools


We should deprecate the openai_builtin_tools in this PR.

So we'll add support for FileSearchToolParam and ComputerToolParam as well?

Note that https://platform.openai.com/docs/guides/tools-file-search also results in a "type": "file_search_call" output item.

Do we still want to allow people to use arbitrary built-in tools we haven't created a class for yet by passing the appropriate JSON?

dmontagu · 2025-05-14T13:19:10Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+class AbstractBuiltinTool(ABC):
+    """A builtin tool that can be used by an agent.
+
+    This class is abstract and cannot be instantiated directly.


I think worth including a sentence here explaining how the code execution works to make use of them — something like "these are passed to the model as part of the ModelRequestParameters" or whatever. (Not sure if that's true, haven't gotten there yet ..). But I imagine it helping someone who is trying to figure out how they are different from normal tools.

dmontagu

LGTM other than tests and docs

Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com>

…1800)

…ther types (#1785)

) Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com>

hyperlint-ai · 2025-05-30T08:39:55Z

PR Change Summary

Enhanced documentation for the Agent class and its output handling, including updates on transport methods and output functions.

Updated transport method descriptions for MCPServerHTTP to reflect the Streamable HTTP transport.
Clarified the use of output functions and their role in agent runs.
Added examples demonstrating the use of output functions and structured outputs.

Modified Files

docs/mcp/client.md
docs/multi-agent-applications.md
docs/output.md
docs/tools.md

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

mattbrandman · 2025-06-06T05:31:40Z

https://github.com/mattbrandman/pydantic-ai/pull/1/files Working on a fork that incorporates the new responses api code interpreter and handles streaming. I'm sure code quality is questionable but feel free to incorporate anything thats useful!

samuelcolvin

LGTM

DouweM · 2025-06-19T17:55:18Z

pydantic_ai_slim/pydantic_ai/models/openai.py

        tools = list(model_settings.get('openai_builtin_tools', [])) + tools
+        tools = self._get_builtin_tools(model_request_parameters) + tools


So we'll add support for FileSearchToolParam and ComputerToolParam as well?

Note that https://platform.openai.com/docs/guides/tools-file-search also results in a "type": "file_search_call" output item.

DouweM · 2025-06-19T18:04:29Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+                        user_location=user_location,
+                    )
+                )
+            elif isinstance(tool, CodeExecutionTool):


I think we'll want an else -> raise "Unsupported built-in tool" here as well (and in OpenAI and Groq, looks like only Google has it currently)

DouweM · 2025-06-19T18:05:34Z

pydantic_ai_slim/pydantic_ai/models/openai.py

+    def _get_builtin_tools(self, model_request_parameters: ModelRequestParameters) -> list[responses.ToolParam]:
+        tools: list[responses.ToolParam] = []
+        for tool in model_request_parameters.builtin_tools:
+            if isinstance(tool, WebSearchTool):


We still need to implement CodeExecutionTool here, right?

The linked commit I have has a basic implementation. Not sure if it would be best to open a PR into this branch or add it after this is merged. There’s unfortunately a bunch of type errors (or were last I checked) from OpenAI where required fields are empty so need a series of guards that look redundant to checkers

No, chat completions doesn't have it.

DouweM · 2025-06-19T18:05:55Z

pydantic_ai_slim/pydantic_ai/models/openai.py

        tools = list(model_settings.get('openai_builtin_tools', [])) + tools
+        tools = self._get_builtin_tools(model_request_parameters) + tools


Do we still want to allow people to use arbitrary built-in tools we haven't created a class for yet by passing the appropriate JSON?

DouweM · 2025-06-19T18:09:05Z

pydantic_ai_slim/pydantic_ai/models/openai.py

@@ -694,6 +721,20 @@ def _get_tools(self, model_request_parameters: ModelRequestParameters) -> list[r
            tools += [self._map_tool_definition(r) for r in model_request_parameters.output_tools]
        return tools

+    def _get_builtin_tools(self, model_request_parameters: ModelRequestParameters) -> list[responses.ToolParam]:


The case I was thinking of that I mentioned on the call this morning is that someone may use OpenAIModel with OpenRouterProvider and an Anthropic model, and it's possible that the built-in tool specifications should then match those for Anthropic rather than those for OpenAI. But from a quick search it doesn't look like OpenRouter currently supports built-in tools (at least there's no doc on it), and when they do they'll probably require a OR-specific specification that they'll translate to the model-specific specs behind the scenes, so we don't have to worry about that for now.

mattbrandman · 2025-06-26T19:11:48Z

besides having tests pass is there anything else holding back this PR seems like it will grow in importance as more builtin tools are introduced?

mattbrandman · 2025-07-01T02:25:00Z

@Kludex not sure if this will be helpful but have a branch based off this with merge conflicts handled and tests running properly. Don't know if I did everything right and also think I should probably have a cleaner one that could be merged into this but my git-foo is failing me late at night will see if I can make a cleaner version mattbrandman#6

DouweM · 2025-07-01T15:27:19Z

@mattbrandman Thanks for creating the #2102 PR! @Kludex is out this week but I'm sure he'll be happy to see the work you've saved him when he's back :)

mattbrandman · 2025-07-01T20:07:10Z

@DouweM no problem! I'm happy to have them merged back into here just couldn't figure out a clean PR strategy since I had to merge main to resolve the conflicts.

Add builtin_tools to Agent

57e568b

Kludex commented May 14, 2025

View reviewed changes

make AbstractBuiltinTool serializable

97ab44b

dmontagu reviewed May 14, 2025

View reviewed changes

Add more work on it

e3dda9d

Kludex marked this pull request as draft May 15, 2025 16:09

Kludex added 2 commits May 23, 2025 08:47

Merge remote-tracking branch 'origin/main' into add-builtin-tools

3ad6d38

Add builtin tools

0b43f65

Kludex force-pushed the add-builtin-tools branch from aa4e770 to 0b43f65 Compare May 24, 2025 14:01

samuelcolvin mentioned this pull request May 26, 2025

Support OpenAI web search tool events in streaming mode #1810

Open

Kludex and others added 14 commits May 26, 2025 14:16

merge

fa7fd11

add more built-in-tools

32324fa

Fix test

f33e568

Add support on Groq

13d7433

Add support for Google

ac85205

Add support for MCP's Streamable HTTP transport (#1716)

c93633f

Timeout for initializing MCP client (#1833)

3a8b640

Require mcp 1.9.0+ (#1840)

360de87

Don't send empty messages to Anthropic (#1027)

cb4e539

Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com>

Add vendor_id and finish_reason to Gemini/Google model responses (#…

4e3769a

…1800)

Fix units of sse_read_timeout timedelta (#1843)

ebb536f

Support functions as output_type, as well as lists of functions and o…

c8bb611

…ther types (#1785)

Enhance Gemini usage tracking to collect comprehensive token data (#1752

6bcc1a8

) Co-authored-by: Marcelo Trylesinski <marcelotryle@gmail.com>

more

97ff651

Kludex added 2 commits May 30, 2025 13:50

merge

1d47e1e

merge

5f89444

Kludex force-pushed the add-builtin-tools branch from 65fa051 to 5f89444 Compare June 1, 2025 11:37

samuelcolvin approved these changes Jun 17, 2025

View reviewed changes

DouweM requested changes Jun 19, 2025

View reviewed changes

Kludex added 2 commits June 20, 2025 10:59

merge

9512987

Pass tests

800a71a

DouweM mentioned this pull request Jun 30, 2025

Support o3-deep-research & o4-mini-deep-research #2086

Open

DouweM mentioned this pull request Jul 1, 2025

Builtin tool #2102

Open

		tools = list(model_settings.get('openai_builtin_tools', [])) + tools
		tools = self._get_builtin_tools(model_request_parameters) + tools

Add builtin_tools to Agent #1722

Are you sure you want to change the base?

Add builtin_tools to Agent #1722

Uh oh!

Conversation

Kludex commented May 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docs Preview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmontagu left a comment

Choose a reason for hiding this comment

Uh oh!

hyperlint-ai bot commented May 30, 2025

PR Change Summary

Uh oh!

mattbrandman commented Jun 6, 2025

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattbrandman commented Jun 26, 2025

Uh oh!

mattbrandman commented Jul 1, 2025

Uh oh!

DouweM commented Jul 1, 2025

Uh oh!

mattbrandman commented Jul 1, 2025

Uh oh!

Uh oh!

Add `builtin_tools` to `Agent` #1722

Add `builtin_tools` to `Agent` #1722

github-actions bot commented May 14, 2025 •

edited

Loading