feat(ai): add vercel ai integration #5858

sabrenner · 2025-06-09T17:52:57Z

What does this PR do?

Adds APM and LLM Observability support for ai@4.0.0 and greater.

DISCLAIMER: Most LOC are from "cassettes" added locally used to mock and play back locally-recorded responses from provider APIs. These are stripped of any sensitive information/headers in the ddapm-test-agent image.

The Vercel AI SDK provides OTel tracing of their operations under the hood. This gives us a nice "in" to patch the tracer used to intercept the startActiveSpan function, and various operations on the underlying span, and translate them into APM and LLM Observability spans.

This integration works by doing exactly that - patching the tracer passed in, and if none is passed in, using a default one and enabling experimental telemetry so that the underlying Vercel AI SDK automatically uses this tracer.

For APM spans:

We are not concerned with setting tags from transformed attributes except the model name and provider from the operation, should the attributes exist.
For LLM Observability, we parse the existing attributes based on a mapping of the operation name to our definition of the different span kinds (workflow, llm, embedding, and tool are applicable)

Additional changes unrelated to the user-facing feature include:

Slightly better LLMObs dev-ex when writing tests by providing a useLlmobs hook that will provide a getEvents function to get APM spans and pre-encoded LLMObs span events. This is just a nice-to-have that can be used in the other integrations as well.
Updates the test agent version to include cassettes for openai through vercel ai

Motivation

Closes #5410

MLOB-2980

github-actions · 2025-06-09T17:53:28Z

Overall package size

Self size: 11.37 MB
Deduped: 110.95 MB
No deduping: 111.34 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.7.0 | 35.02 MB | 35.02 MB | | @datadog/native-appsec | 10.0.1 | 20.3 MB | 20.3 MB | | @datadog/native-iast-taint-tracking | 4.0.0 | 11.72 MB | 11.73 MB | | @datadog/pprof | 5.9.0 | 9.77 MB | 10.14 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.5.3 | 2.95 MB | 5.6 MB | | @datadog/wasm-js-rewriter | 4.0.1 | 2.85 MB | 3.58 MB | | @datadog/native-metrics | 3.1.1 | 1.02 MB | 1.43 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | jsonpath-plus | 10.3.0 | 617.18 kB | 1.08 MB | | import-in-the-middle | 1.14.2 | 122.36 kB | 850.93 kB | | lru-cache | 10.4.3 | 804.3 kB | 804.3 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 7.0.5 | 63.38 kB | 63.38 kB | | istanbul-lib-coverage | 3.2.2 | 34.37 kB | 34.37 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | shell-quote | 1.8.3 | 23.74 kB | 23.74 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | mutexify | 1.4.0 | 5.71 kB | 8.74 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.4 | 3.96 kB | 3.96 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2025-06-09T17:54:03Z

Codecov Report

Attention: Patch coverage is 94.87179% with 10 lines in your changes missing coverage. Please review.

Project coverage is 83.23%. Comparing base (a216c23) to head (f9cfc5c).

Files with missing lines	Patch %	Lines
packages/dd-trace/src/llmobs/plugins/ai/index.js	94.80%	8 Missing ⚠️
...ages/datadog-instrumentations/src/helpers/hooks.js	0.00%	1 Missing ⚠️
packages/dd-trace/src/plugins/index.js	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5858      +/-   ##
==========================================
+ Coverage   82.81%   83.23%   +0.42%     
==========================================
  Files         476      478       +2     
  Lines       19664    19857     +193     
==========================================
+ Hits        16284    16528     +244     
+ Misses       3380     3329      -51

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pr-commenter · 2025-06-09T17:59:29Z

Benchmarks

Benchmark execution time: 2025-07-23 18:48:50

Comparing candidate commit f9cfc5c in PR branch sabrenner/vercel-ai-sdk-integration with baseline commit a216c23 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1272 metrics, 51 unstable metrics.

datadog-datadog-prod-us1 · 2025-06-09T18:07:29Z

Datadog Report

Branch report: sabrenner/vercel-ai-sdk-integration
Commit report: 9b1dfdb
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 1257 Passed, 0 Skipped, 20m 11.47s Total Time

packages/dd-trace/src/id.js

packages/datadog-instrumentations/src/helpers/hooks.js

packages/datadog-instrumentations/src/vercel-ai.js

packages/dd-trace/src/id.js

packages/dd-trace/src/llmobs/tagger.js

packages/dd-trace/src/llmobs/plugins/vercel-ai.js

sabrenner · 2025-06-11T18:58:43Z

packages/datadog-instrumentations/src/ai.js

+const noopTracer = {
+  startActiveSpan () {
+    const fn = arguments[arguments.length - 1]
+
+    const span = {
+      end () {},
+      setAttributes () { return this },
+      addEvent () { return this },
+      recordException () { return this },
+      setStatus () { return this }
+    }
+
+    return fn(span)
+  }
+}


i guess this could be extracted out into an otel noop tracer that could be shared

Why even define a noop tracer that will ultimately be patched in the first place? Why not just returning a fake span directly?

i've cleaned this up, although it's still local to the instrumentation, but i'm gonna resolve for now anyways since it's only used in this instrumentation. if we need a no-op dummy otel tracer for other instrumentations down the line i think we can refactor then.

Why not just returning a fake span directly?

mostly because we could be patching an actual otel-compatible tracer that someone is already using. i agree if we were just concerned about a dummy tracer/spans, then yeah we could patch in-place with respect to the dummy tracer, but i did it this way bc someone could actually be using a real tracer. lmk if that answers your question!

…r/vercel-ai-sdk-integration

sabrenner · 2025-07-15T18:10:01Z

for reviewers (when i open this up): currently this is all one PR - apm + llmobs. just the way i did the poc + clean up, but if this pr is too big i'm happy to separate it out!

rochdev · 2025-06-12T21:19:21Z

packages/datadog-instrumentations/src/ai.js

+const noopTracer = {
+  startActiveSpan () {
+    const fn = arguments[arguments.length - 1]
+
+    const span = {
+      end () {},
+      setAttributes () { return this },
+      addEvent () { return this },
+      recordException () { return this },
+      setStatus () { return this }
+    }
+
+    return fn(span)
+  }
+}


Why even define a noop tracer that will ultimately be patched in the first place? Why not just returning a fake span directly?

packages/dd-trace/src/llmobs/tagger.js

packages/datadog-instrumentations/src/vercel-ai.js

packages/datadog-instrumentations/src/ai.js

rochdev · 2025-07-21T15:42:19Z

packages/datadog-plugin-ai/test/index.spec.js

+        /**
+         * Resolves the following error:
+         *
+         * Error [ERR_REQUIRE_ESM]: require() of ES Module  from ... not supported.


Not sure I understand this comment. If it's supported in CommonJS (which should be true according to the instrumentation) it should means we can also import it here no?

i think it's not supported in commonjs for the restrictions specified in the condition below (early version 4 of vercel ai sdk + node < 22 not supported. i get this when just running a dummy script requiring ai with ai@4.0.0.0 and Node 20

if that makes sense, i can update the comment so it's not as confusing 😅

I guess my concern is that if it's supported, it should work in tests, and if it's not supported, then we should change the range. But I may still not be grasping the issue correctly 😅

packages/dd-trace/src/llmobs/plugins/ai.js

packages/dd-trace/test/llmobs/util.js

.github/actions/testagent-llmobs/start/action.yml

This reverts commit 39ccf68.

This reverts commit ac4071c.

…ached" This reverts commit 72eca83.

This reverts commit 20c3c63.

This reverts commit 32d75a6.

…r/vercel-ai-sdk-integration

datadog-datadog-prod-us1 · 2025-07-23T17:51:17Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f9cfc5c | Was this helpful? Give us feedback!}

rochdev · 2025-07-23T20:51:32Z

packages/dd-trace/test/llmobs/util.js

  })

  after(() => {
-    LLMObsSpanWriter.prototype.append.restore()
+    process.removeAllListeners()


Is this safe? I think the test runner might have listeners on the process.

Scalahansolo · 2025-07-23T20:57:02Z

Hey hey crew. Fwiw... I've been silently following this PR in wait to try and weave this into the Motion codebase to get our stack onto LLMObservability. Currently we don't use dd-trace, and just export spans / traces through otel. I imagine once this lands I'll need to pull in dd-trace for LLM Observability.

Would it be helpful at all if I were to test this (somehow?) and report back here?

sabrenner added 2 commits June 9, 2025 13:44

add vercel ai integration with otel processing

94a2b68

add some typedocs and comments

0ed7d4c

sabrenner added the semver-minor label Jun 9, 2025

fix tagger test

233beb8

sabrenner commented Jun 9, 2025

View reviewed changes

packages/dd-trace/src/id.js Outdated Show resolved Hide resolved

rochdev reviewed Jun 9, 2025

View reviewed changes

sabrenner added 3 commits June 9, 2025 15:56

rename to 'ai'

90f4f13

try doing with a custom tracer

a4014cc

change up implementation slightly

407d6dc

sabrenner commented Jun 11, 2025

View reviewed changes

sabrenner added 16 commits June 17, 2025 09:59

codeowners

16c3d28

undo id changes

fbff2a3

get rid of otel span start/end publishes

d963226

revert llmobs tagger test change

067c596

add better noop default tracer and esm support

24f37a8

delete util file

563d56f

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

7e906a2

…r/vercel-ai-sdk-integration

add initial test skeleton

47056b9

fix duplicate wrapping

2da477f

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

a43db02

…r/vercel-ai-sdk-integration

simplify patching

cc9b031

apm tests

2015808

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

2ce1798

…r/vercel-ai-sdk-integration

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

5a5a23c

…r/vercel-ai-sdk-integration

write some tests

16c5da4

add rest of llmobs tests

0c2df21

some self review

95ebae1

sabrenner changed the title ~~wip(vercel-ai): add vercel ai integration~~ feat(ai): add vercel ai integration Jul 14, 2025

sabrenner marked this pull request as ready for review July 19, 2025 02:56

sabrenner requested review from a team as code owners July 19, 2025 02:56

rochdev reviewed Jul 21, 2025

View reviewed changes

sabrenner added 7 commits July 21, 2025 12:26

address some review comments

d19e31b

do not stub for tests, instead use dummy test agent

c03fc82

move cassettes to local directory to fix tests

20c3c63

configurable flush interval for tests

0f1acf6

use separate image for ai tests that have local cassettes attached

72eca83

use different port

ac4071c

move test flush interval back local

39ccf68

sabrenner commented Jul 21, 2025

View reviewed changes

.github/actions/testagent-llmobs/start/action.yml Outdated Show resolved Hide resolved

sabrenner added 10 commits July 21, 2025 16:23

change in esm test

32d75a6

Revert "move test flush interval back local"

d1fbc64

This reverts commit 39ccf68.

Revert "use different port"

b705691

This reverts commit ac4071c.

Revert "use separate image for ai tests that have local cassettes att…

b28bd92

…ached" This reverts commit 72eca83.

Revert "move cassettes to local directory to fix tests"

bec9312

This reverts commit 20c3c63.

Revert "change in esm test"

b642965

This reverts commit 32d75a6.

remove env var from llmobs workflow

bbbfffb

fix type hint for test util

85abd5d

Merge branch 'master' of github.com:DataDog/dd-trace-js into sabrenne…

8d1662e

…r/vercel-ai-sdk-integration

add test cassettes

2bd92ee

re-trigger ci

f9cfc5c

rochdev reviewed Jul 23, 2025

View reviewed changes

feat(ai): add vercel ai integration #5858

Are you sure you want to change the base?

feat(ai): add vercel ai integration #5858

Conversation

sabrenner commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Uh oh!

github-actions bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

codecov bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

datadog-datadog-prod-us1 bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sabrenner Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sabrenner commented Jul 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1 bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Scalahansolo commented Jul 23, 2025

Uh oh!

Uh oh!

sabrenner commented Jun 9, 2025 •

edited

Loading

github-actions bot commented Jun 9, 2025 •

edited

Loading

codecov bot commented Jun 9, 2025 •

edited

Loading

pr-commenter bot commented Jun 9, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Jun 9, 2025 •

edited

Loading

sabrenner Jul 21, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Jul 23, 2025 •

edited

Loading