Skip to content

Commit 86c8f70

Browse files
antonpirkers1gr1dbuenaflorlucas-zimerman
authored
Add spec for "AI Agents" insights module to develop docs. (#14147)
<!-- Use this checklist to make sure your PR is ready for merge. You may delete any sections you don't need. --> ## DESCRIBE YOUR PR Describing the spans and attributes that are needed for data showing up in the "AI Agents" insights module. Preview: https://develop-docs-git-antonpirker-developai-agents-module.sentry.dev/sdk/telemetry/traces/modules/ai-agents/ ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. - [x] Urgent deadline (GA date, etc.): GA date: 2025-06-26 - [ ] Other deadline: <!-- ENTER DATE HERE --> - [ ] None: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've added an urgent due date to it. Thanks in advance for your help! ## PRE-MERGE CHECKLIST *Make sure you've checked the following before merging your changes:* - [x] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) ## LEGAL BOILERPLATE <!-- Sentry employees and contractors can delete or ignore this section. --> Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms. ## EXTRA RESOURCES - [Sentry Docs contributor guide](https://docs.sentry.io/contributing/) --------- Co-authored-by: Sigrid Huemer <32902192+s1gr1d@users.noreply.github.com> Co-authored-by: Giancarlo Buenaflor <giancarlo_buenaflor@yahoo.com> Co-authored-by: LucasZF <lucas-zimerman1@hotmail.com>
1 parent 0f850ed commit 86c8f70

File tree

1 file changed

+142
-0
lines changed

1 file changed

+142
-0
lines changed

‎develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
---
2+
title: AI Agents Module
3+
---
4+
5+
The AI Agents module is agnostic to the library used. The SDK will instrument existing AI agents in certain frameworks or libraries (at the time of writing those are `openai-agents` in Python and `Vercel AI` in Javascript). You may need to manually annotate spans for other libraries.
6+
7+
## Spans Conventions
8+
9+
For your AI agents data to show up in the Sentry [AI Agents Insights](https://sentry.io/orgredirect/organizations/:orgslug/insights/agents/) a couple of different spans must be created and have well defined names and data attributes. See below.
10+
11+
We try to follow the [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) for Generative AI as close as possible. Being 100% compatible is not yet possible, because OpenTelemetry as "Span Events" which Sentry does not support. The input/output to/from an AI model is stored in span events in OpenTelemetry, as this is not possible in Sentry we add this data onto span attributes as a list.
12+
13+
### Invoke Agent Span
14+
15+
Describes AI agent invocation.
16+
17+
- The spans `op` MUST be `"gen_ai.invoke_agent"`.
18+
- The span `name` SHOULD be `"invoke_agent {gen_ai.agent.name}"`.
19+
- The `gen_ai.operation.name` attribute MUST be `"invoke_agent"`.
20+
- The `gen_ai.agent.name` attribute SHOULD be set to the agents name. (e.g. `"Weather Agent"`)
21+
- All [Common Span Attributes](#common-span-attributes) SHOULD be set (all `required` common attributes MUST be set).
22+
23+
Additional attributes on the span:
24+
25+
| Data Attribute | Type | Requirement Level | Description | Example |
26+
| :------------------------------------- | :----- | :---------------- | :-------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------- |
27+
| `gen_ai.request.available_tools` | string | optional | List of dictionaries describing the available tools. **[0]** | `"[{\"name\": \"random_number\", \"description\": \"...\"}, {\"name\": \"query_db\", \"description\": \"...\"}]"` |
28+
| `gen_ai.request.frequency_penalty` | float | optional | Model configuration parameter. | `0.5` |
29+
| `gen_ai.request.max_tokens` | int | optional | Model configuration parameter. | `500` |
30+
| `gen_ai.request.messages` | string | optional | List of dictionaries describing the messages (prompts) sent to the LLM **[0]**, **[1]** | `"[{\"role\": \"system\", \"content\": [{...}]}, {\"role\": \"system\", \"content\": [{...}]}]"` |
31+
| `gen_ai.request.presence_penalty` | float | optional | Model configuration parameter. | `0.5` |
32+
| `gen_ai.request.temperature` | float | optional | Model configuration parameter. | `0.1` |
33+
| `gen_ai.request.top_p` | float | optional | Model configuration parameter. | `0.7` |
34+
| `gen_ai.response.tool_calls` | string | optional | The tool calls in the model’s response. **[0]** | `"[{\"name\": \"random_number\", \"type\": \"function_call\", \"arguments\": \"...\"}]"` |
35+
| `gen_ai.response.text` | string | optional | The text representation of the model’s responses. **[0]** | `"[\"The weather in Paris is rainy\", \"The weather in London is sunny\"]"` |
36+
| `gen_ai.usage.input_tokens.cached` | int | optional | The number of cached tokens used in the AI input (prompt) | `50` |
37+
| `gen_ai.usage.input_tokens` | int | optional | The number of tokens used in the AI input (prompt). | `10` |
38+
| `gen_ai.usage.output_tokens.reasoning` | int | optional | The number of tokens used for reasoning. | `30` |
39+
| `gen_ai.usage.output_tokens` | int | optional | The number of tokens used in the AI response. | `100` |
40+
| `gen_ai.usage.total_tokens` | int | optional | The total number of tokens used to process the prompt. (input and output) | `190` |
41+
42+
- **[0]:** As span attributes only allow primitive data types (like `int`, `float`, `boolean`, `string`) this needs to be a stringified version of a list of dictionaries. Do NOT set `[{"foo": "bar"}]` but rather the string `"[{\"foo\": \"bar\"}]"`.
43+
- **[1]:** Each item in the list of messages has the format `{role:"", content:""}` where `role` can be `"user"`, `"assistant"`, or `"system"` and `content` can either be a string or a list of dictionaries.
44+
45+
### AI Client Span
46+
47+
This span represents a request to an AI model or service that generates a response or requests a tool call based on the input prompt.
48+
49+
- The span `op` MUST be `"gen_ai.{gen_ai.operation.name}"`. (e.g. `"gen_ai.chat"`)
50+
- The span `name` SHOULD be `{gen_ai.operation.name} {gen_ai.request.model}"`. (e.g. `"chat o3-mini"`)
51+
- All [Common Span Attributes](#common-span-attributes) SHOULD be set (all `required` common attributes MUST be set).
52+
53+
Additional attributes on the span:
54+
55+
| Data Attribute | Type | Requirement Level | Description | Example |
56+
| :------------------------------------- | :----- | :---------------- | :-------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------- |
57+
| `gen_ai.request.available_tools` | string | optional | List of dictionaries describing the available tools. **[0]** | `"[{\"name\": \"random_number\", \"description\": \"...\"}, {\"name\": \"query_db\", \"description\": \"...\"}]"` |
58+
| `gen_ai.request.frequency_penalty` | float | optional | Model configuration parameter. | `0.5` |
59+
| `gen_ai.request.max_tokens` | int | optional | Model configuration parameter. | `500` |
60+
| `gen_ai.request.messages` | string | optional | List of dictionaries describing the messages (prompts) sent to the LLM **[0]**, **[1]** | `"[{\"role\": \"system\", \"content\": [{...}]}, {\"role\": \"system\", \"content\": [{...}]}]"` |
61+
| `gen_ai.request.presence_penalty` | float | optional | Model configuration parameter. | `0.5` |
62+
| `gen_ai.request.temperature` | float | optional | Model configuration parameter. | `0.1` |
63+
| `gen_ai.request.top_p` | float | optional | Model configuration parameter. | `0.7` |
64+
| `gen_ai.response.tool_calls` | string | optional | The tool calls in the model’s response. **[0]** | `"[{\"name\": \"random_number\", \"type\": \"function_call\", \"arguments\": \"...\"}]"` |
65+
| `gen_ai.response.text` | string | optional | The text representation of the model’s responses. **[0]** | `"[\"The weather in Paris is rainy\", \"The weather in London is sunny\"]"` |
66+
| `gen_ai.usage.input_tokens.cached` | int | optional | The number of cached tokens used in the AI input (prompt) | `50` |
67+
| `gen_ai.usage.input_tokens` | int | optional | The number of tokens used in the AI input (prompt). | `10` |
68+
| `gen_ai.usage.output_tokens.reasoning` | int | optional | The number of tokens used for reasoning. | `30` |
69+
| `gen_ai.usage.output_tokens` | int | optional | The number of tokens used in the AI response. | `100` |
70+
| `gen_ai.usage.total_tokens` | int | optional | The total number of tokens used to process the prompt. (input and output) | `190` |
71+
72+
- **[0]:** As span attributes only allow primitive data types this needs to be a stringified version of a list of dictionaries. Do NOT set `[{"foo": "bar"}]` but rather the string `"[{\"foo\": \"bar\"}]"`.
73+
- **[1]:** Each item in the list has the format `{role:"", content:""}` where `role` can be `"user"`, `"assistant"`, or `"system"` and `content` can either be a string or a list of dictionaries.
74+
75+
### Execute Tool Span
76+
77+
Describes a tool execution.
78+
79+
- The spans `op` MUST be `"gen_ai.execute_tool"`.
80+
- The spans `name` SHOULD be `"gen_ai.execute_tool {gen_ai.tool.name}"`. (e.g. `"gen_ai.execute_tool query_database"`)
81+
- The `gen_ai.tool.name` attribute SHOULD be set to the name of the tool. (e.g. `"query_database"`)
82+
- All [Common Span Attributes](#common-span-attributes) SHOULD be set (all `required` common attributes MUST be set).
83+
84+
Additional attributes on the span:
85+
86+
| Data Attribute | Type | Requirement Level | Description | Example |
87+
| :------------------------ | :----- | :---------------- | :--------------------------------------------------- | :----------------------------------------- |
88+
| `gen_ai.tool.description` | string | optional | Description of the tool executed. | `"Tool returning a random number"` |
89+
| `gen_ai.tool.input` | string | optional | Input that was given to the executed tool as string. | `"{\"max\":10}"` |
90+
| `gen_ai.tool.name:` | string | optional | Name of the tool executed. | `"random_number"` |
91+
| `gen_ai.tool.output` | string | optional | The output from the tool. | `"7"` |
92+
| `gen_ai.tool.type` | string | optional | The type of the tools. | `"function"`; `"extension"`; `"datastore"` |
93+
94+
### Handoff Span
95+
96+
A span that describes the handoff from one agent to another agent.
97+
98+
- The spans `op` MUST be `"gen_ai.handoff"`.
99+
- The spans `name` SHOULD be `"handoff from {from_agent} to {from_agent}"`.
100+
- All [Common Span Attributes](#common-span-attributes) SHOULD be set (all `required` common attributes MUST be set).
101+
102+
## Common Span Attributes
103+
104+
Some attributes are common to all AI Agents spans:
105+
106+
| Data Attribute | Type | Requirement Level | Description | Example |
107+
| :---------------------- | :----- | :---------------- | :--------------------------------------------------------------------------------------- | :---------------- |
108+
| `gen_ai.system` | string | required | The Generative AI product as identified by the client or server instrumentation. **[0]** | `"openai"` |
109+
| `gen_ai.request.model` | string | required | The name of the AI model a request is being made to. | `"o3-mini"` |
110+
| `gen_ai.operation.name` | string | optional | The name of the operation being performed. **[1]** | `"chat"` |
111+
| `gen_ai.agent.name` | string | optional | The name of the agent this span belongs to. | `"Weather Agent"` |
112+
113+
**[0]** Well defined values for data attribute `gen_ai.system`:
114+
115+
| Value | Description |
116+
| :------------------ | :-------------------------------- |
117+
| `"anthropic"` | Anthropic |
118+
| `"aws.bedrock"` | AWS Bedrock |
119+
| `"az.ai.inference"` | Azure AI Inference |
120+
| `"az.ai.openai"` | Azure OpenAI |
121+
| `"cohere"` | Cohere |
122+
| `"deepseek"` | DeepSeek |
123+
| `"gcp.gemini"` | Gemini |
124+
| `"gcp.gen_ai"` | Any Google generative AI endpoint |
125+
| `"gcp.vertex_ai"` | Vertex AI |
126+
| `"groq"` | Groq |
127+
| `"ibm.watsonx.ai"` | IBM Watsonx AI |
128+
| `"mistral_ai"` | Mistral AI |
129+
| `"openai"` | OpenAI |
130+
| `"perplexity"` | Perplexity |
131+
| `"xai"` | xAI |
132+
133+
**[1]** Well defined values for data attribute `gen_ai.operation.name`:
134+
135+
| Value | Description |
136+
| :------------------- | :---------------------------------------------------------------------- |
137+
| `"chat"` | Chat completion operation such as OpenAI Chat API |
138+
| `"create_agent"` | Create GenAI agent |
139+
| `"embeddings"` | Embeddings operation such as OpenAI Create embeddings API |
140+
| `"execute_tool"` | Execute a tool |
141+
| `"generate_content"` | Multimodal content generation operation such as Gemini Generate Content |
142+
| `"invoke_agent"` | Invoke GenAI agent |

0 commit comments

Comments
 (0)