Skip to content

feat: predefined query for extracting tool calls #7438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
May 28, 2025

Conversation

tinuademargaret
Copy link
Contributor

@tinuademargaret tinuademargaret commented May 6, 2025

Issue: Fixes #7158

Description:
This PR introduces a predefined query get_called_tools for querying LLM tool calls and their associated input/output messages from span data.

Example usage:

from phoenix.trace.dsl.helpers import get_called_tools

tools_df = get_called_tools(client)

# Example output:
# context.span_id  | tool_call
# ----------------|------------------
# span_1          | ["multiply", "add"]  
# span_2          | ["add"]            

This has been tested to work with the top llm providers and framework auto-instrumentations noted in the issue

Copy link
Contributor

github-actions bot commented May 6, 2025

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@tinuademargaret tinuademargaret marked this pull request as ready for review May 9, 2025 21:22
@tinuademargaret tinuademargaret requested a review from a team as a code owner May 9, 2025 21:22
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label May 9, 2025
@tinuademargaret
Copy link
Contributor Author

I have read the CLA Document and I hereby sign the CLA

github-actions bot added a commit that referenced this pull request May 9, 2025
@tinuademargaret
Copy link
Contributor Author

I was wondering if the parameters of the tool should also be returned by the query, maybe have that as an option?

@axiomofjoy axiomofjoy requested a review from RogerHYang May 13, 2025 16:18
@Jgilhuly
Copy link
Contributor

Thank you for this @tinuademargaret! Doing some final testing today with a range of providers

@Jgilhuly
Copy link
Contributor

@RogerHYang - tested from my side with our top 10 providers. All working fine!

One note here: the helper exports the user input under the column header input, but our tool calling template looks for question as the header. We should be able to immediately put the output of this helper into the template, but I'd actually argue that input is the better header name. I'd rather we update the template to match this, than the other way around. Lmk your thoughts there

Copy link
Contributor

@RogerHYang RogerHYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution, @tinuademargaret!

I've added a flag to include function arguments to the result by default.

@github-project-automation github-project-automation bot moved this from 📘 Todo to 👍 Approved in phoenix May 28, 2025
@RogerHYang RogerHYang merged commit 8be7147 into Arize-ai:main May 28, 2025
37 of 38 checks passed
@github-project-automation github-project-automation bot moved this from 👍 Approved to ✅ Done in phoenix May 28, 2025
efontan-dialpad pushed a commit to gluru/phoenix that referenced this pull request May 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:L This PR changes 100-499 lines, ignoring generated files.
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Add predefined queries for extracting tool calls
3 participants