feat: pydanticai instrumentation #1639

davidgmonical · 2025-05-16T20:00:10Z

resolves: #1348

Adds instrumentation for PydanticAI. Captures and converts gen_ai attributes from running an Agent with instrument=True.

mikeldking

This is awesome - though I wonder if we should re-think this category of integration. It's not an instrumentor after all.

On the js side we just do openinference-xxxx

Note you are missing :

the changes to the base readme
release please manifest changes
symlinking support for a single interactive install

mikeldking · 2025-05-21T00:21:16Z

python/instrumentation/openinference-instrumentation-pydanticai/MIGRATION.md

@@ -0,0 +1 @@
+# Migrations


I'm not sure you need this?

mikeldking · 2025-05-21T00:21:47Z

python/instrumentation/openinference-instrumentation-pydanticai/README.md

@@ -0,0 +1,3 @@
+# OpenInference PydanticAI Instrumentation
+
+Python auto-instrumentation library for PydanticAI.


can you flesh this out more? it gets scaffolded into pypi and is important for various reasons like search.

mikeldking · 2025-05-21T00:22:21Z

python/instrumentation/openinference-instrumentation-pydanticai/examples/pydantic.py

@@ -0,0 +1,45 @@
+import os


nit: name it according to the use case. it's easier to grep for

can you also add a more interesting example?

mikeldking · 2025-05-21T00:23:28Z

python/instrumentation/openinference-instrumentation-pydanticai/examples/pydantic.py

+# Launch Phoenix app
+session = px.launch_app()


I would probably avoid this pattern - it encourages bad practices. Just assume phoenix is running.

mikeldking · 2025-05-21T00:27:02Z

python/instrumentation/openinference-instrumentation-pydanticai/pyproject.toml

+build-backend = "hatchling.build"
+
+[project]
+name = "openinference-instrumentation-pydanticai"


I think if this is not actually an instrumentor, it might be worth naming this opeininference-pydanticai rather than instrumentation since it's not an instrumentor

though if this just makes the symlinking confusing I think it's okay. Defer to @axiomofjoy

Would also use pydantic-ai everywhere rather than pydanticai to match the package name

mikeldking · 2025-05-21T00:31:36Z

python/instrumentation/openinference-instrumentation-pydanticai/pyproject.toml

+  "pydantic>=2.10.0",
+  "pydantic-ai>=0.2.0",


is the pydantic necessary here?

mikeldking · 2025-05-21T00:34:12Z

python/instrumentation/openinference-instrumentation-pydanticai/examples/pydantic.py

+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+from pydantic import BaseModel


I get a circular import on this line:

Traceback (most recent call last): File "/Users/mikeldking/work/openinference/python/instrumentation/openinference-instrumentation-pydanticai/./examples/pydantic.py", line 3, in <module> import phoenix as px File "/Users/mikeldking/work/phoenix/src/phoenix/__init__.py", line 10, in <module> from .session.client import Client File "/Users/mikeldking/work/phoenix/src/phoenix/session/client.py", line 31, in <module> from phoenix.db.insertion.dataset import DatasetKeys File "/Users/mikeldking/work/phoenix/src/phoenix/db/__init__.py", line 1, in <module> from .engines import get_printable_db_url File "/Users/mikeldking/work/phoenix/src/phoenix/db/engines.py", line 21, in <module> from phoenix.db.helpers import SupportedSQLDialect File "/Users/mikeldking/work/phoenix/src/phoenix/db/helpers.py", line 23, in <module> from phoenix.db import models File "/Users/mikeldking/work/phoenix/src/phoenix/db/models.py", line 48, in <module> from phoenix.db.types.annotation_configs import ( File "/Users/mikeldking/work/phoenix/src/phoenix/db/types/annotation_configs.py", line 4, in <module> from pydantic import AfterValidator, Field, RootModel, model_validator File "/Users/mikeldking/work/openinference/python/instrumentation/openinference-instrumentation-pydanticai/examples/pydantic.py", line 8, in <module> from pydantic import BaseModel ImportError: cannot import name 'BaseModel' from partially initialized module 'pydantic' (most likely due to a circular import) (/Users/mikeldking/work/openinference/python/instrumentation/openinference-instrumentation-pydanticai/examples/pydantic.py)

should be fixed once I rename the script

mikeldking · 2025-05-21T00:37:06Z

...nce-instrumentation-pydanticai/src/openinference/instrumentation/pydanticai/span_exporter.py

+        result = self._base_exporter.force_flush(timeout_millis)
+        return bool(result)


why not just return the base force_flush?

mikeldking · 2025-05-21T00:43:06Z

...trumentation-pydanticai/src/openinference/instrumentation/pydanticai/semantic_conventions.py

+    FUNCTION = "function"
+
+
+class GenAICommonAttributes:


is there no package that has these experimental attributes?

Good call out - looks like I can get most of them from the opentelemetry-python package.

There seem to be some left out, specifically the body fields for events (the docs link an issue to migrate them which might be why). I will keep those defined in our own constants but use the imported ones wherever I can

Agree here. The more we can leverage types/ enums/ constants from OTel or Pydantic itself, the better.

axiomofjoy · 2025-05-21T22:45:36Z

python/instrumentation/openinference-instrumentation-pydanticai/pyproject.toml

+build-backend = "hatchling.build"
+
+[project]
+name = "openinference-instrumentation-pydanticai"


Would also use pydantic-ai everywhere rather than pydanticai to match the package name

axiomofjoy · 2025-05-21T22:58:03Z

...strumentation-pydanticai/tests/openinference/instrumentation/pydanticai/test_instrumentor.py

+        assert attributes.get(SpanAttributes.LLM_TOKEN_COUNT_PROMPT) == 58
+        assert attributes.get(SpanAttributes.LLM_TOKEN_COUNT_COMPLETION) == 22
+        assert attributes.get(SpanAttributes.LLM_TOKEN_COUNT_TOTAL) == 80


Let's not hard code these to any particular recording of the cassette so we can re-run and re-record the tests easily, e.g., just check for int and check that the numbers add up.

axiomofjoy · 2025-05-21T23:00:34Z

...nference-instrumentation-pydanticai/src/openinference/instrumentation/pydanticai/__init__.py

+from openinference.instrumentation.pydanticai.span_exporter import OpenInferenceSpanExporter
+from openinference.instrumentation.pydanticai.utils import is_openinference_span
+from openinference.instrumentation.pydanticai.version import __version__
+
+__all__ = [
+    "OpenInferenceSpanExporter",
+    "is_openinference_span",
+    "__version__",
+]


Suggested change

from openinference.instrumentation.pydanticai.span_exporter import OpenInferenceSpanExporter

from openinference.instrumentation.pydanticai.utils import is_openinference_span

from openinference.instrumentation.pydanticai.version import __version__

__all__ = [

"OpenInferenceSpanExporter",

"is_openinference_span",

"__version__",

]

from openinference.instrumentation.pydanticai.span_exporter import OpenInferenceSpanExporter

from openinference.instrumentation.pydanticai.version import __version__

__all__ = [

"OpenInferenceSpanExporter",

"__version__",

]

This looks like a helper function we are using in tests only rather than something we want to expose to end users?

Yeah this was kind of copying the vercel js implementation. I assumed it's because you end up with some spans that don't map to openinference ones (something I saw as well in pydantic). I was thinking it is a convenience function for a user to block those from getting to phoenix, but probably need more context (maybe @Parker-Stafford can chime in) on what that was for in vercel

axiomofjoy · 2025-05-21T23:02:11Z

...eninference-instrumentation-pydanticai/src/openinference/instrumentation/pydanticai/utils.py

+def is_openinference_span(span: ReadableSpan) -> bool:
+    """Check if a span is an OpenInference span."""
+    if span.attributes is None:
+        return False
+    return SpanAttributes.OPENINFERENCE_SPAN_KIND in span.attributes


This looks like something we only use in tests. Can I be in the test file rather than shipped with the package?

axiomofjoy · 2025-05-21T23:02:39Z

...nce-instrumentation-pydanticai/src/openinference/instrumentation/pydanticai/span_exporter.py

+from openinference.instrumentation.pydanticai.utils import SpanFilter, should_export_span
+
+
+class OpenInferenceSpanExporter(SpanExporter):


Curious why use an exporter rather than a span processor?

axiomofjoy · 2025-05-21T23:05:21Z

...trumentation-pydanticai/src/openinference/instrumentation/pydanticai/semantic_conventions.py

+class OpenInferenceAttributesExtractor:
+    """Extracts OpenInference attributes from GenAI attributes."""
+
+    def __init__(self) -> None:
+        pass


Strikes me that all of the methods on this class are pure functions and the class has no state. Maybe they can just be functions?

Yeah makes sense

axiomofjoy · 2025-05-21T23:07:54Z

...trumentation-pydanticai/src/openinference/instrumentation/pydanticai/semantic_conventions.py

+    FUNCTION = "function"
+
+
+class GenAICommonAttributes:


Agree here. The more we can leverage types/ enums/ constants from OTel or Pydantic itself, the better.

…openinference into david/pydantic-instrumentation

mikeldking · 2025-05-22T04:36:56Z

That trace looks pretty great - I wish we could figure out the tool spans!

mikeldking · 2025-05-22T04:37:39Z

python/instrumentation/openinference-instrumentation-pydantic-ai/README.md

+Start Phoenix in the background as a collector. By default, it listens on `http://localhost:6006`. You can visit the app via a browser at the same address. (Phoenix does not send data over the internet. It only operates locally on your machine.)
+
+```shell
+python -m phoenix.server.main serve


Suggested change

python -m phoenix.server.main serve

phoenix serve

mikeldking · 2025-05-22T04:38:39Z

...umentation-pydantic-ai/src/openinference/instrumentation/pydantic_ai/semantic_conventions.py

+import logging
+from typing import Any, Dict, Iterator, List, Mapping, Optional, Tuple, cast
+
+from opentelemetry.semconv._incubating.attributes.gen_ai_attributes import (


mikeldking · 2025-05-22T04:43:27Z

python/instrumentation/openinference-instrumentation-pydantic-ai/pyproject.toml

+dependencies = [
+  "opentelemetry-api",
+  "opentelemetry-instrumentation",
+  "opentelemetry-semantic-conventions",


I think we need a minimum version here. I get an import error:

ImportError: cannot import name 'GEN_AI_REQUEST_SEED' from 'opentelemetry.semconv._incubating.attributes.gen_ai_attributes' (/Users/mikeldking/anaconda3/envs/phoenix/lib/python3.12/site-packages/opentelemetry/semconv/_incubating/attributes/gen_ai_attributes.py). Did you mean: 'GEN_AI_REQUEST_MODEL'?

mikeldking · 2025-05-22T04:46:49Z

I hit a road block with the import of the conventions. Will dogfood tomorrow.

davidgmonical added 8 commits May 1, 2025 19:25

wip

a5fcbea

rework attribute mapping plus test

a4cd44a

add files

8ec54a6

clean up

309e92c

fix structure and clean

e921980

change package name

36d7256

passing tox

2ca2764

merge main

38bc449

davidgmonical requested a review from a team as a code owner May 16, 2025 20:00

github-project-automation bot added this to Instrumentation May 16, 2025

dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label May 16, 2025

davidgmonical changed the title ~~feat: David/pydantic instrumentation~~ feat: pydanticai instrumentation May 16, 2025

caroger requested a review from axiomofjoy May 20, 2025 17:42

add a requirments file

6d24561

mikeldking reviewed May 21, 2025

View reviewed changes

axiomofjoy reviewed May 21, 2025

View reviewed changes

davidgmonical added 3 commits May 21, 2025 20:04

rename package, import gen_ai attributes, fix tests

49b0ad1

Merge branch 'david/pydantic-instrumentation' of github.com:Arize-ai/…

755b9f7

…openinference into david/pydantic-instrumentation

merge

4783464

mikeldking reviewed May 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: pydanticai instrumentation #1639

feat: pydanticai instrumentation #1639

davidgmonical commented May 16, 2025

mikeldking left a comment

mikeldking May 21, 2025

mikeldking May 21, 2025

mikeldking May 21, 2025

mikeldking May 21, 2025

mikeldking May 21, 2025

mikeldking May 21, 2025

mikeldking May 21, 2025

axiomofjoy May 21, 2025

mikeldking May 21, 2025

mikeldking May 21, 2025

davidgmonical May 22, 2025

mikeldking May 21, 2025

mikeldking May 21, 2025

davidgmonical May 21, 2025

axiomofjoy May 21, 2025

axiomofjoy May 21, 2025

axiomofjoy May 21, 2025

axiomofjoy May 21, 2025

davidgmonical May 22, 2025

axiomofjoy May 21, 2025

axiomofjoy May 21, 2025

axiomofjoy May 21, 2025

davidgmonical May 22, 2025

axiomofjoy May 21, 2025

mikeldking commented May 22, 2025

mikeldking May 22, 2025

mikeldking May 22, 2025

mikeldking May 22, 2025

mikeldking commented May 22, 2025

		@@ -0,0 +1,3 @@
		# OpenInference PydanticAI Instrumentation

		Python auto-instrumentation library for PydanticAI.

		result = self._base_exporter.force_flush(timeout_millis)
		return bool(result)

		from openinference.instrumentation.pydanticai.utils import SpanFilter, should_export_span


		class OpenInferenceSpanExporter(SpanExporter):

feat: pydanticai instrumentation #1639

Are you sure you want to change the base?

feat: pydanticai instrumentation #1639

Conversation

davidgmonical commented May 16, 2025

mikeldking left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikeldking commented May 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikeldking commented May 22, 2025