[WIP] Cap and update versions in CI #4906

bouwkast · 2024-11-19T02:11:49Z

What does this PR do?

Adds outdated.js and latests.json to be used to cap versions pulled in during testing via mocha.js.

To generate or update latests.json:
node scripts/outdated.js fix

To just validate latests.json:
node scripts/outdated.js check

Idea is to use mocha.js to try and enforce these capped versions.

TODO:

Review/Remove the register/install_plugin_modules from Bryan
- We don't want to cap the instrumented version at this time which I think these do
Create script to automatically call node scripts/outdated.js fix and make a PR
Test this as I don't know how :)

Motivation

We'd like to cap the versions pulled in during testing to keep CI more stable.
We'd like to create a GitHub Action that runs consistently to keep these updated and get better visibility of unsupported versions.

Plugin Checklist

Additional Notes

github-actions · 2024-11-19T02:12:35Z

Overall package size

Self size: 8.83 MB
Deduped: 95.03 MB
No deduping: 95.54 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.4.0 | 29.44 MB | 29.44 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.5.1 | 9.79 MB | 10.17 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.8.0 | 2.6 MB | 2.74 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 835.4 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

pr-commenter · 2024-12-23T22:40:37Z

Benchmarks

Benchmark execution time: 2025-02-27 21:11:30

Comparing candidate commit 611eafd in PR branch steven/cap-and-update with baseline commit 2c9ab13 in branch master.

Found 72 performance improvements and 8 performance regressions! Performance is the same for 808 metrics, 15 unstable metrics.

scenario:appsec-iast-no-vulnerability-control-18

🟩 cpu_user_time [-93.927ms; -81.720ms] or [-16.097%; -14.005%]
🟩 execution_time [-100.115ms; -76.096ms] or [-14.729%; -11.195%]
🟩 instructions [-182.4M instructions; -160.1M instructions] or [-13.427%; -11.785%]
🟩 max_rss_usage [-5.555MB; -4.530MB] or [-7.024%; -5.728%]

scenario:appsec-iast-no-vulnerability-control-20

🟩 cpu_user_time [-105.584ms; -95.714ms] or [-18.544%; -16.811%]
🟩 execution_time [-108.868ms; -96.591ms] or [-16.810%; -14.914%]
🟩 instructions [-164.4M instructions; -145.8M instructions] or [-13.608%; -12.067%]
🟩 max_rss_usage [-4.382MB; -3.916MB] or [-5.716%; -5.109%]

scenario:appsec-iast-no-vulnerability-control-22

🟩 cpu_user_time [-94.538ms; -83.902ms] or [-18.496%; -16.415%]
🟩 execution_time [-98.833ms; -83.412ms] or [-16.563%; -13.979%]
🟩 instructions [-162.7M instructions; -140.7M instructions] or [-13.106%; -11.332%]
🟩 max_rss_usage [-6.865MB; -6.163MB] or [-7.940%; -7.128%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-18

🟩 cpu_user_time [-146.310ms; -124.212ms] or [-11.039%; -9.372%]
🟩 execution_time [-154.233ms; -122.829ms] or [-10.539%; -8.393%]
🟩 instructions [-222.5M instructions; -204.6M instructions] or [-8.388%; -7.716%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

🟩 cpu_user_time [-171.243ms; -150.683ms] or [-12.411%; -10.921%]
🟩 execution_time [-176.022ms; -159.001ms] or [-11.631%; -10.506%]
🟩 instructions [-187.7M instructions; -164.7M instructions] or [-7.204%; -6.321%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-22

🟩 cpu_user_time [-117.317ms; -99.646ms] or [-11.303%; -9.600%]
🟩 execution_time [-122.877ms; -102.547ms] or [-10.495%; -8.758%]
🟩 instructions [-186.4M instructions; -159.0M instructions] or [-7.285%; -6.215%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-18

🟩 cpu_user_time [-119.010ms; -105.990ms] or [-9.316%; -8.297%]
🟩 execution_time [-127.729ms; -105.043ms] or [-9.045%; -7.438%]
🟩 instructions [-217.7M instructions; -185.1M instructions] or [-8.775%; -7.462%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

🟩 cpu_user_time [-154.073ms; -139.944ms] or [-11.532%; -10.474%]
🟩 execution_time [-165.110ms; -149.838ms] or [-11.197%; -10.161%]
🟩 instructions [-196.4M instructions; -173.8M instructions] or [-8.050%; -7.121%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-22

🟩 cpu_user_time [-113.604ms; -95.885ms] or [-11.283%; -9.524%]
🟩 execution_time [-113.440ms; -97.365ms] or [-9.980%; -8.566%]
🟩 instructions [-180.4M instructions; -147.8M instructions] or [-7.558%; -6.193%]

scenario:appsec-iast-with-vulnerability-control-18

🟩 cpu_user_time [-108.011ms; -84.805ms] or [-13.188%; -10.355%]
🟩 execution_time [-137.754ms; -112.848ms] or [-10.379%; -8.502%]
🟩 instructions [-205.5M instructions; -186.1M instructions] or [-8.570%; -7.763%]

scenario:appsec-iast-with-vulnerability-control-20

🟩 cpu_user_time [-111.652ms; -95.136ms] or [-14.087%; -12.003%]
🟩 execution_time [-131.650ms; -115.507ms] or [-12.269%; -10.764%]
🟩 instructions [-173.2M instructions; -153.6M instructions] or [-8.986%; -7.969%]

scenario:appsec-iast-with-vulnerability-control-22

🟩 cpu_user_time [-94.998ms; -75.989ms] or [-13.603%; -10.881%]
🟩 execution_time [-109.188ms; -91.269ms] or [-11.302%; -9.447%]
🟩 instructions [-156.9M instructions; -131.8M instructions] or [-8.085%; -6.790%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-18

🟩 cpu_user_time [-373.648ms; -332.388ms] or [-19.665%; -17.494%]
🟩 execution_time [-492.655ms; -400.410ms] or [-17.606%; -14.309%]
🟩 instructions [-525.7M instructions; -486.4M instructions] or [-11.733%; -10.856%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

🟩 cpu_user_time [-324.603ms; -290.027ms] or [-17.243%; -15.406%]
🟩 execution_time [-389.631ms; -341.329ms] or [-15.905%; -13.934%]
🟩 instructions [-381.0M instructions; -349.8M instructions] or [-9.632%; -8.845%]
🟩 max_rss_usage [-11.939MB; -9.492MB] or [-9.332%; -7.420%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-22

🟩 cpu_user_time [-200.460ms; -171.078ms] or [-13.557%; -11.570%]
🟩 execution_time [-241.658ms; -195.355ms] or [-12.094%; -9.777%]
🟩 instructions [-376.8M instructions; -339.9M instructions] or [-9.670%; -8.723%]
🟩 max_rss_usage [-11.815MB; -8.328MB] or [-9.259%; -6.526%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-18

🟩 cpu_user_time [-322.228ms; -288.756ms] or [-18.063%; -16.187%]
🟩 execution_time [-437.636ms; -359.124ms] or [-16.850%; -13.827%]
🟩 instructions [-390.7M instructions; -338.9M instructions] or [-9.753%; -8.459%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20

🟩 cpu_user_time [-273.327ms; -244.887ms] or [-15.410%; -13.806%]
🟩 execution_time [-328.660ms; -284.316ms] or [-14.517%; -12.558%]
🟩 instructions [-299.6M instructions; -251.4M instructions] or [-8.484%; -7.121%]
🟩 max_rss_usage [-11.526MB; -8.197MB] or [-9.037%; -6.427%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-22

🟩 cpu_user_time [-140.482ms; -111.107ms] or [-10.530%; -8.328%]
🟩 execution_time [-169.013ms; -126.415ms] or [-9.542%; -7.137%]
🟩 instructions [-275.6M instructions; -234.3M instructions] or [-7.957%; -6.766%]

scenario:plugin-graphql-control-18

🟥 cpu_user_time [+16.974ms; +24.417ms] or [+5.647%; +8.123%]
🟥 execution_time [+20.007ms; +24.468ms] or [+5.735%; +7.014%]

scenario:plugin-graphql-with-async-hooks-18

🟥 cpu_user_time [+23.254ms; +35.781ms] or [+6.177%; +9.504%]
🟥 execution_time [+27.414ms; +39.607ms] or [+6.379%; +9.217%]

scenario:plugin-q-control-20

🟥 cpu_user_time [+60.460ms; +70.335ms] or [+12.103%; +14.080%]
🟥 execution_time [+60.722ms; +68.921ms] or [+10.593%; +12.024%]

scenario:plugin-q-control-22

🟥 cpu_user_time [+37.347ms; +46.578ms] or [+8.280%; +10.327%]
🟥 execution_time [+37.490ms; +44.868ms] or [+7.195%; +8.611%]

scenario:plugin-q-with-tracer-18

🟩 cpu_user_time [-147.882ms; -127.854ms] or [-16.513%; -14.277%]
🟩 execution_time [-155.065ms; -134.985ms] or [-15.648%; -13.622%]
🟩 instructions [-309.8M instructions; -269.9M instructions] or [-16.475%; -14.354%]
🟩 max_rss_usage [-9.804MB; -9.315MB] or [-6.534%; -6.209%]

scenario:plugin-q-with-tracer-20

🟩 cpu_user_time [-116.228ms; -97.629ms] or [-13.591%; -11.416%]
🟩 execution_time [-122.264ms; -103.811ms] or [-12.864%; -10.923%]
🟩 instructions [-298.8M instructions; -266.0M instructions] or [-17.476%; -15.555%]
🟩 max_rss_usage [-10.389MB; -9.955MB] or [-6.969%; -6.678%]

scenario:plugin-q-with-tracer-22

🟩 cpu_user_time [-178.094ms; -159.180ms] or [-20.317%; -18.160%]
🟩 execution_time [-189.201ms; -170.006ms] or [-19.424%; -17.454%]
🟩 instructions [-316.9M instructions; -267.0M instructions] or [-17.793%; -14.987%]
🟩 max_rss_usage [-17.051MB; -15.597MB] or [-10.857%; -9.931%]

Opting to not use later version as it has too many changes focused on ranges/matrices.

This also adds some logging as it is excruciatingly slow and I want to know what it is doing.

Taken from Bryan

codecov · 2025-02-26T21:29:19Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.01%. Comparing base (2c9ab13) to head (611eafd).
Report is 15 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4906      +/-   ##
==========================================
- Coverage   80.46%   73.01%   -7.45%     
==========================================
  Files         491      428      -63     
  Lines       21850    19372    -2478     
==========================================
- Hits        17581    14144    -3437     
- Misses       4269     5228     +959

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

datadog-datadog-prod-us1 · 2025-02-26T21:34:19Z

Datadog Report

Branch report: steven/cap-and-update
Commit report: aca1e66
Test service: dd-trace-js-integration-tests

❌ 51 Failed (0 Known Flaky), 464 Passed, 0 Skipped, 7m 9.78s Total Time

❌ Failed Tests (51)

This report shows up to 5 failed tests.

cucumber@latest commonJS correctly calculates test code owners when working directory is not repository root - integration-tests/cucumber/cucumber.spec.js - Details
Expand for error
```
 Cannot read properties of undefined (reading 'content')
```
cucumber@latest commonJS correctly calculates test code owners when working directory is not repository root - integration-tests/cucumber/cucumber.spec.js - Details
Expand for error
```
 Cannot read properties of undefined (reading 'content')
```
cucumber@latest commonJS reporting via agentless intelligent test runner can report git metadata - integration-tests/cucumber/cucumber.spec.js - Details
Expand for error
```
 Timeout
```
cucumber@latest commonJS reporting via agentless intelligent test runner can report git metadata - integration-tests/cucumber/cucumber.spec.js - Details
Expand for error
```
 Timeout
```
cucumber@latest commonJS reporting via agentless intelligent test runner can skip suites received by the intelligent test runner API and still reports code coverage - integration-tests/cucumber/cucumber.spec.js - Details
Expand for error
```
 Timeout
```

BridgeAR · 2025-05-28T23:57:15Z

I very much like the overall idea here. Is that still worked upon?

BridgeAR · 2025-05-28T23:58:29Z

Seems like this is doing something similar to #4819?

bouwkast force-pushed the steven/cap-and-update branch from 2663b15 to cff1165 Compare December 24, 2024 00:52

bouwkast added 7 commits February 26, 2025 07:45

Add versioning.js

aeb4a5d

Add outdated.js

2b6695a

Opting to not use later version as it has too many changes focused on ranges/matrices.

Update outdated to generate latests.json

193d84d

This also adds some logging as it is excruciatingly slow and I want to know what it is doing.

Hack mocha.js to use latests instead of max

fd5e1be

Update register and install_plugin_modules

a88cd4b

Taken from Bryan

Attempt to fix issue installing latest always

353fffb

Attempt to validate semver ranges

345ed6b

bouwkast force-pushed the steven/cap-and-update branch from e8869a6 to 345ed6b Compare February 26, 2025 21:27

bouwkast added 2 commits February 27, 2025 14:44

Attempt to install correct version when given a range

080d7fa

Attempt to better handle and parse @ and version ranges

611eafd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Cap and update versions in CI #4906

[WIP] Cap and update versions in CI #4906

Uh oh!

bouwkast commented Nov 19, 2024

Uh oh!

github-actions bot commented Nov 19, 2024 •

edited

Loading

Uh oh!

pr-commenter bot commented Dec 23, 2024 •

edited

Loading

Uh oh!

codecov bot commented Feb 26, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented Feb 26, 2025 •

edited

Loading

Uh oh!

BridgeAR commented May 28, 2025

Uh oh!

BridgeAR commented May 28, 2025

Uh oh!

Uh oh!

[WIP] Cap and update versions in CI #4906

Are you sure you want to change the base?

[WIP] Cap and update versions in CI #4906

Uh oh!

Conversation

bouwkast commented Nov 19, 2024

What does this PR do?

Motivation

Plugin Checklist

Additional Notes

Uh oh!

github-actions bot commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Uh oh!

pr-commenter bot commented Dec 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:appsec-iast-no-vulnerability-control-18

scenario:appsec-iast-no-vulnerability-control-20

scenario:appsec-iast-no-vulnerability-control-22

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-18

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-22

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-18

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-22

scenario:appsec-iast-with-vulnerability-control-18

scenario:appsec-iast-with-vulnerability-control-20

scenario:appsec-iast-with-vulnerability-control-22

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-18

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-22

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-18

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-22

scenario:plugin-graphql-control-18

scenario:plugin-graphql-with-async-hooks-18

scenario:plugin-q-control-20

scenario:plugin-q-control-22

scenario:plugin-q-with-tracer-18

scenario:plugin-q-with-tracer-20

scenario:plugin-q-with-tracer-22

Uh oh!

codecov bot commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-datadog-prod-us1 bot commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

❌ Failed Tests (51)

Uh oh!

BridgeAR commented May 28, 2025

Uh oh!

BridgeAR commented May 28, 2025

Uh oh!

Uh oh!

github-actions bot commented Nov 19, 2024 •

edited

Loading

pr-commenter bot commented Dec 23, 2024 •

edited

Loading

codecov bot commented Feb 26, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Feb 26, 2025 •

edited

Loading