Skip to content

Conversation

@apiarian-datadog
Copy link
Contributor

What does this PR do?

Adds FIPS support. This is intended for use with bottlecap for now, so the changes here are very focused on bottlecap usage: DataDog/datadog-lambda-extension#644 .

Additional Notes

I'm not a rust expert, please advise better, cleaner, more correct ways to do this.

How to test the change?

Built and deployed bottlecap in all of our various self-monitoring stacks.

@apiarian-datadog apiarian-datadog requested review from a team as code owners April 16, 2025 14:33
@apiarian-datadog apiarian-datadog changed the title Aleksandr.pasechnik/svls 6242 fips features for bottlecap feat: [SVLS-6272] fips features for bottlecap Apr 16, 2025
@pr-commenter
Copy link

pr-commenter bot commented Apr 16, 2025

Benchmarks

Comparison

Benchmark execution time: 2025-04-24 16:58:13

Comparing candidate commit 1a60690 in PR branch aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap with baseline commit d433dd2 in branch main.

Found 17 performance improvements and 2 performance regressions! Performance is the same for 33 metrics, 2 unstable metrics.

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟩 execution_time [-4.730µs; -4.553µs] or [-5.711%; -5.497%]
  • 🟩 throughput [+703233.924op/s; +731436.060op/s] or [+5.824%; +6.058%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟩 execution_time [-5.892µs; -5.764µs] or [-7.643%; -7.477%]
  • 🟩 throughput [+1049380.754op/s; +1072188.321op/s] or [+8.089%; +8.265%]

scenario:credit_card/is_card_number/378282246310005

  • 🟩 execution_time [-6.143µs; -5.980µs] or [-8.327%; -8.106%]
  • 🟩 throughput [+1197329.274op/s; +1229362.010op/s] or [+8.832%; +9.069%]

scenario:credit_card/is_card_number/x371413321323331

  • 🟩 execution_time [-523.971ns; -504.202ns] or [-7.977%; -7.676%]
  • 🟩 throughput [+12680468.773op/s; +13150081.365op/s] or [+8.328%; +8.637%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

  • 🟩 execution_time [-534.808ns; -511.117ns] or [-8.120%; -7.760%]
  • 🟩 throughput [+12798505.765op/s; +13357326.620op/s] or [+8.428%; +8.796%]

scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...

  • 🟥 execution_time [+9.565µs; +9.700µs] or [+4.582%; +4.647%]
  • 🟥 throughput [-212759.272op/s; -209883.641op/s] or [-4.441%; -4.381%]

scenario:normalization/normalize_name/normalize_name/bad-name

  • 🟩 execution_time [-1.056µs; -1.040µs] or [-5.673%; -5.584%]
  • 🟩 throughput [+3177640.529op/s; +3229428.966op/s] or [+5.916%; +6.013%]

scenario:normalization/normalize_name/normalize_name/good

  • 🟩 execution_time [-753.146ns; -739.313ns] or [-6.807%; -6.682%]
  • 🟩 throughput [+6477033.379op/s; +6592969.492op/s] or [+7.167%; +7.295%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

  • 🟩 execution_time [-36.974µs; -36.577µs] or [-7.326%; -7.247%]
  • 🟩 throughput [+154850.905op/s; +156644.988op/s] or [+7.815%; +7.906%]

scenario:sql/obfuscate_sql_string

  • 🟩 execution_time [-2.832µs; -2.753µs] or [-4.135%; -4.020%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 3.896µs 3.913µs ± 0.003µs 3.914µs ± 0.002µs 3.915µs 3.917µs 3.919µs 3.922µs 0.21% -1.050 6.751 0.07% 0.000µs 1 200
credit_card/is_card_number/ throughput 254996499.784op/s 255530016.567op/s ± 177597.829op/s 255524901.242op/s ± 111951.411op/s 255649238.045op/s 255809810.105op/s 255882963.826op/s 256645470.991op/s 0.44% 1.066 6.851 0.07% 12558.063op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 77.038µs 78.181µs ± 0.565µs 78.109µs ± 0.372µs 78.505µs 79.196µs 79.485µs 79.942µs 2.35% 0.393 -0.307 0.72% 0.040µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 12509140.642op/s 12791531.527op/s ± 92131.597op/s 12802671.737op/s ± 60980.692op/s 12857667.096op/s 12924564.135op/s 12958243.401op/s 12980561.257op/s 1.39% -0.360 -0.349 0.72% 6514.688op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 70.689µs 71.262µs ± 0.273µs 71.248µs ± 0.208µs 71.473µs 71.672µs 71.939µs 71.991µs 1.04% 0.158 -0.495 0.38% 0.019µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 13890549.953op/s 14032977.762op/s ± 53766.129op/s 14035566.925op/s ± 40983.967op/s 14074801.200op/s 14121070.685op/s 14134264.706op/s 14146479.674op/s 0.79% -0.141 -0.510 0.38% 3801.839op/s 1 200
credit_card/is_card_number/37828224631 execution_time 3.892µs 3.914µs ± 0.003µs 3.914µs ± 0.001µs 3.915µs 3.918µs 3.921µs 3.921µs 0.19% -2.314 19.857 0.07% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 255014893.828op/s 255487365.589op/s ± 176816.574op/s 255491824.763op/s ± 85872.992op/s 255575496.701op/s 255686263.445op/s 255940227.676op/s 256907814.923op/s 0.55% 2.348 20.163 0.07% 12502.820op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 67.046µs 67.710µs ± 0.357µs 67.687µs ± 0.234µs 67.937µs 68.349µs 68.776µs 68.931µs 1.84% 0.615 0.772 0.53% 0.025µs 1 200
credit_card/is_card_number/378282246310005 throughput 14507208.269op/s 14769378.837op/s ± 77567.946op/s 14773855.498op/s ± 51000.260op/s 14821632.549op/s 14891497.416op/s 14905926.390op/s 14915078.782op/s 0.96% -0.577 0.687 0.52% 5484.882op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 51.753µs 51.850µs ± 0.033µs 51.847µs ± 0.017µs 51.865µs 51.907µs 51.961µs 51.966µs 0.23% 0.559 1.453 0.06% 0.002µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 19243257.407op/s 19286347.195op/s ± 12328.323op/s 19287541.213op/s ± 6378.159op/s 19293166.360op/s 19305714.879op/s 19313610.825op/s 19322717.719op/s 0.18% -0.553 1.445 0.06% 871.744op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 6.028µs 6.054µs ± 0.027µs 6.042µs ± 0.006µs 6.073µs 6.112µs 6.148µs 6.197µs 2.56% 2.033 5.059 0.45% 0.002µs 1 200
credit_card/is_card_number/x371413321323331 throughput 161381228.054op/s 165173505.203op/s ± 736105.045op/s 165513959.896op/s ± 175019.872op/s 165646291.856op/s 165781493.297op/s 165851523.514op/s 165891436.148op/s 0.23% -1.994 4.812 0.44% 52050.487op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 3.894µs 3.914µs ± 0.003µs 3.914µs ± 0.001µs 3.915µs 3.918µs 3.920µs 3.921µs 0.18% -1.565 10.715 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 255056449.201op/s 255519324.370op/s ± 181655.890op/s 255518106.447op/s ± 92749.003op/s 255607565.999op/s 255783194.489op/s 255937907.975op/s 256779103.968op/s 0.49% 1.587 10.877 0.07% 12845.011op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 65.989µs 66.375µs ± 0.283µs 66.275µs ± 0.139µs 66.587µs 66.880µs 67.226µs 67.334µs 1.60% 1.079 0.513 0.43% 0.020µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 14851285.521op/s 15066073.575op/s ± 63957.724op/s 15088684.242op/s ± 31695.763op/s 15113224.073op/s 15136917.509op/s 15147897.669op/s 15154043.867op/s 0.43% -1.062 0.458 0.42% 4522.494op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 59.508µs 59.655µs ± 0.062µs 59.647µs ± 0.041µs 59.692µs 59.766µs 59.808µs 59.858µs 0.35% 0.444 0.188 0.10% 0.004µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 16706138.233op/s 16763183.218op/s ± 17399.075op/s 16765314.241op/s ± 11523.262op/s 16775898.486op/s 16788821.839op/s 16798937.165op/s 16804565.801op/s 0.23% -0.438 0.179 0.10% 1230.300op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 3.895µs 3.913µs ± 0.003µs 3.913µs ± 0.001µs 3.914µs 3.916µs 3.918µs 3.919µs 0.15% -1.578 8.153 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 255196217.148op/s 255582797.409op/s ± 178690.370op/s 255572443.826op/s ± 95025.820op/s 255664775.786op/s 255883045.940op/s 255975102.653op/s 256756501.653op/s 0.46% 1.594 8.273 0.07% 12635.317op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 56.286µs 56.519µs ± 0.110µs 56.518µs ± 0.074µs 56.586µs 56.719µs 56.800µs 56.857µs 0.60% 0.395 0.178 0.19% 0.008µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 17588065.569op/s 17693078.637op/s ± 34404.863op/s 17693371.780op/s ± 23173.548op/s 17717947.300op/s 17747714.524op/s 17762900.092op/s 17766506.582op/s 0.41% -0.383 0.162 0.19% 2432.791op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 51.759µs 51.844µs ± 0.037µs 51.841µs ± 0.014µs 51.854µs 51.896µs 51.948µs 52.090µs 0.48% 2.573 13.974 0.07% 0.003µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 19197722.215op/s 19288487.592op/s ± 13576.951op/s 19289837.149op/s ± 5135.125op/s 19295041.174op/s 19302165.896op/s 19317866.363op/s 19320328.850op/s 0.16% -2.553 13.834 0.07% 960.035op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 6.028µs 6.063µs ± 0.032µs 6.044µs ± 0.010µs 6.079µs 6.116µs 6.150µs 6.155µs 1.84% 1.032 0.077 0.53% 0.002µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 162474247.881op/s 164928296.784op/s ± 870910.261op/s 165464571.068op/s ± 269660.914op/s 165602510.313op/s 165736177.924op/s 165858878.918op/s 165903240.774op/s 0.27% -1.016 0.028 0.53% 61582.655op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [3.913µs; 3.914µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/ throughput [255505403.216op/s; 255554629.918op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [78.103µs; 78.259µs] or [-0.100%; +0.100%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [12778762.974op/s; 12804300.080op/s] or [-0.100%; +0.100%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [71.224µs; 71.300µs] or [-0.053%; +0.053%] None None None
credit_card/is_card_number/ 378282246310005 throughput [14025526.293op/s; 14040429.230op/s] or [-0.053%; +0.053%] None None None
credit_card/is_card_number/37828224631 execution_time [3.914µs; 3.914µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/37828224631 throughput [255462860.512op/s; 255511870.665op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/378282246310005 execution_time [67.660µs; 67.759µs] or [-0.073%; +0.073%] None None None
credit_card/is_card_number/378282246310005 throughput [14758628.666op/s; 14780129.008op/s] or [-0.073%; +0.073%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [51.846µs; 51.855µs] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [19284638.608op/s; 19288055.782op/s] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/x371413321323331 execution_time [6.051µs; 6.058µs] or [-0.062%; +0.062%] None None None
credit_card/is_card_number/x371413321323331 throughput [165071488.123op/s; 165275522.282op/s] or [-0.062%; +0.062%] None None None
credit_card/is_card_number_no_luhn/ execution_time [3.913µs; 3.914µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ throughput [255494148.610op/s; 255544500.129op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [66.336µs; 66.415µs] or [-0.059%; +0.059%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [15057209.649op/s; 15074937.500op/s] or [-0.059%; +0.059%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [59.646µs; 59.663µs] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [16760771.873op/s; 16765594.562op/s] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [3.912µs; 3.913µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [255558032.642op/s; 255607562.176op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [56.504µs; 56.535µs] or [-0.027%; +0.027%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [17688310.453op/s; 17697846.820op/s] or [-0.027%; +0.027%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [51.839µs; 51.849µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [19286605.957op/s; 19290369.227op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [6.059µs; 6.068µs] or [-0.074%; +0.074%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [164807596.998op/s; 165048996.570op/s] or [-0.073%; +0.073%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 462.419µs 467.910µs ± 1.365µs 468.069µs ± 0.898µs 468.897µs 469.609µs 470.134µs 473.289µs 1.12% -0.478 1.552 0.29% 0.097µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 2112872.073op/s 2137179.098op/s ± 6245.407op/s 2136435.709op/s ± 4089.389op/s 2141036.256op/s 2148006.495op/s 2150965.569op/s 2162542.130op/s 1.22% 0.507 1.563 0.29% 441.617op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 450.202µs 451.205µs ± 0.961µs 451.096µs ± 0.280µs 451.391µs 451.946µs 457.260µs 458.666µs 1.68% 5.662 39.040 0.21% 0.068µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2180237.153op/s 2216296.342op/s ± 4665.447op/s 2216821.184op/s ± 1377.731op/s 2218125.591op/s 2220345.327op/s 2221081.832op/s 2221226.866op/s 0.20% -5.605 38.482 0.21% 329.897op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 179.840µs 180.353µs ± 0.421µs 180.291µs ± 0.187µs 180.519µs 180.738µs 180.962µs 185.231µs 2.74% 7.860 88.261 0.23% 0.030µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5398655.840op/s 5544703.927op/s ± 12713.803op/s 5546599.076op/s ± 5741.434op/s 5551455.903op/s 5555815.505op/s 5558500.439op/s 5560500.483op/s 0.25% -7.663 85.208 0.23% 899.002op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 38.991µs 39.165µs ± 0.066µs 39.163µs ± 0.039µs 39.206µs 39.277µs 39.351µs 39.386µs 0.57% 0.245 0.679 0.17% 0.005µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 25389419.392op/s 25532754.429op/s ± 43024.949op/s 25534584.234op/s ± 25151.835op/s 25557278.868op/s 25601027.029op/s 25637804.671op/s 25647084.846op/s 0.44% -0.232 0.667 0.17% 3042.323op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 48.265µs 48.374µs ± 0.046µs 48.369µs ± 0.028µs 48.397µs 48.453µs 48.508µs 48.563µs 0.40% 0.979 1.517 0.10% 0.003µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 20591731.484op/s 20672388.266op/s ± 19737.016op/s 20674447.464op/s ± 11829.517op/s 20686116.161op/s 20699324.320op/s 20704465.974op/s 20718869.428op/s 0.21% -0.972 1.495 0.10% 1395.618op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [467.721µs; 468.100µs] or [-0.040%; +0.040%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [2136313.545op/s; 2138044.651op/s] or [-0.040%; +0.040%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [451.072µs; 451.338µs] or [-0.030%; +0.030%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2215649.756op/s; 2216942.928op/s] or [-0.029%; +0.029%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [180.295µs; 180.412µs] or [-0.032%; +0.032%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5542941.916op/s; 5546465.938op/s] or [-0.032%; +0.032%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [39.156µs; 39.175µs] or [-0.023%; +0.023%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [25526791.585op/s; 25538717.273op/s] or [-0.023%; +0.023%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [48.367µs; 48.380µs] or [-0.013%; +0.013%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [20669652.906op/s; 20675123.627op/s] or [-0.013%; +0.013%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.314µs 2.380µs ± 0.018µs 2.379µs ± 0.007µs 2.386µs 2.411µs 2.422µs 2.428µs 2.08% -0.512 2.238 0.75% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.377µs; 2.382µs] or [-0.105%; +0.105%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
ip_address/quantize_peer_ip_address_benchmark execution_time 4.985µs 5.046µs ± 0.024µs 5.051µs ± 0.019µs 5.066µs 5.078µs 5.080µs 5.082µs 0.61% -0.436 -0.848 0.48% 0.002µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark execution_time [5.042µs; 5.049µs] or [-0.067%; +0.067%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 32.244µs 32.964µs ± 0.940µs 32.376µs ± 0.059µs 34.294µs 34.382µs 35.359µs 35.669µs 10.17% 0.981 -0.795 2.84% 0.066µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [32.834µs; 33.095µs] or [-0.395%; +0.395%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.208µs 3.237µs ± 1.485µs 3.038µs ± 0.022µs 3.055µs 3.689µs 14.151µs 15.997µs 426.55% 7.474 56.865 45.76% 0.105µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [3.032µs; 3.443µs] or [-6.358%; +6.358%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 17.674µs 25.991µs ± 11.061µs 17.992µs ± 0.172µs 35.285µs 44.072µs 54.392µs 97.134µs 439.87% 1.943 7.750 42.45% 0.782µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [24.458µs; 27.524µs] or [-5.898%; +5.898%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 217.509µs 218.363µs ± 0.459µs 218.346µs ± 0.346µs 218.669µs 219.143µs 219.431µs 220.063µs 0.79% 0.486 0.131 0.21% 0.032µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 4544163.295op/s 4579545.460op/s ± 9607.535op/s 4579897.157op/s ± 7237.363op/s 4587628.406op/s 4593728.999op/s 4595998.424op/s 4597503.661op/s 0.38% -0.474 0.104 0.21% 679.355op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 17.448µs 17.571µs ± 0.049µs 17.570µs ± 0.033µs 17.599µs 17.652µs 17.693µs 17.715µs 0.83% 0.352 0.133 0.28% 0.003µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 56448566.653op/s 56913777.841op/s ± 157714.689op/s 56914804.950op/s ± 106298.345op/s 57027275.079op/s 57148072.596op/s 57279765.602op/s 57312420.031op/s 0.70% -0.335 0.120 0.28% 11152.113op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 10.268µs 10.319µs ± 0.023µs 10.314µs ± 0.017µs 10.336µs 10.355µs 10.369µs 10.376µs 0.59% -0.015 -0.545 0.22% 0.002µs 1 200
normalization/normalize_name/normalize_name/good throughput 96379230.938op/s 96912923.393op/s ± 218286.273op/s 96952185.929op/s ± 160227.849op/s 97047065.322op/s 97307505.049op/s 97342283.751op/s 97388935.725op/s 0.45% 0.024 -0.545 0.22% 15435.170op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [218.300µs; 218.427µs] or [-0.029%; +0.029%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [4578213.948op/s; 4580876.972op/s] or [-0.029%; +0.029%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [17.564µs; 17.577µs] or [-0.038%; +0.038%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [56891920.102op/s; 56935635.580op/s] or [-0.038%; +0.038%] None None None
normalization/normalize_name/normalize_name/good execution_time [10.315µs; 10.322µs] or [-0.031%; +0.031%] None None None
normalization/normalize_name/normalize_name/good throughput [96882671.015op/s; 96943175.771op/s] or [-0.031%; +0.031%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 72.774ms 73.323ms ± 0.361ms 73.256ms ± 0.218ms 73.552ms 73.822ms 74.157ms 76.110ms 3.90% 2.587 16.188 0.49% 0.026ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [73.272ms; 73.373ms] or [-0.068%; +0.068%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 246.427ns 256.600ns ± 14.199ns 249.382ns ± 1.669ns 256.770ns 288.488ns 302.774ns 305.115ns 22.35% 1.837 2.464 5.52% 1.004ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [254.632ns; 258.568ns] or [-0.767%; +0.767%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 65.449µs 65.690µs ± 0.224µs 65.667µs ± 0.061µs 65.730µs 65.861µs 66.151µs 68.371µs 4.12% 8.803 101.193 0.34% 0.016µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [65.659µs; 65.721µs] or [-0.047%; +0.047%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 5.967ms 5.976ms ± 0.011ms 5.975ms ± 0.003ms 5.978ms 5.983ms 6.021ms 6.100ms 2.09% 7.875 78.704 0.18% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [5.975ms; 5.978ms] or [-0.026%; +0.026%] None None None

Group 13

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 1a60690 1745513181 aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 147.161µs 148.126µs ± 0.320µs 148.074µs ± 0.154µs 148.252µs 148.670µs 148.953µs 150.387µs 1.56% 2.059 12.037 0.22% 0.023µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [148.082µs; 148.171µs] or [-0.030%; +0.030%] None None None

Baseline

Omitted due to size.

@r1viollet
Copy link
Contributor

r1viollet commented Apr 16, 2025

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 76.93 MB 65.97 MB --14.25% (-10.96 MB) 💪
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so.debug 23.72 MB 20.29 MB --14.43% (-3.42 MB) 💪
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 7.70 MB 8.03 MB +4.22% (+333.51 KB) ⚠️
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 7.63 MB 8.00 MB +4.86% (+380.34 KB) ⚠️
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so.debug 22.32 MB 22.73 MB +1.82% (+416.42 KB) ⚠️
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 71.28 MB 72.72 MB +2.01% (+1.43 MB) ⚠️
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 16.73 MB 16.73 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 62.20 KB 62.20 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 112.78 MB 112.77 MB -0% (-8.00 KB) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 632.36 MB 632.34 MB -0% (-22.32 KB) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 4.99 MB 4.99 MB +.03% (+2.00 KB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 62.20 KB 62.20 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 15.87 MB 15.87 MB +.04% (+8.00 KB) 🔍
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 26.40 MB 26.41 MB +.06% (+16.93 KB) 🔍
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 14.14 MB 14.15 MB +.07% (+11.00 KB) 🔍
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 63.16 KB 63.16 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 114.58 MB 114.71 MB +.10% (+128.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 624.22 MB 624.95 MB +.11% (+743.56 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 3.77 MB 3.78 MB +.03% (+1.50 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 63.16 KB 63.16 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 16.48 MB 16.49 MB +.04% (+8.00 KB) 🔍
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 24.32 MB 24.33 MB +.05% (+13.91 KB) 🔍
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 66.30 MB 57.52 MB --13.23% (-8.77 MB) 💪
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.19 MB 8.43 MB +2.92% (+244.96 KB) ⚠️
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so.debug 22.90 MB 18.72 MB --18.25% (-4.17 MB) 💪
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 67.27 MB 68.60 MB +1.97% (+1.33 MB) ⚠️
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 8.07 MB 8.43 MB +4.36% (+361.42 KB) ⚠️
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so.debug 20.55 MB 20.90 MB +1.70% (+359.03 KB) ⚠️

@apiarian-datadog apiarian-datadog force-pushed the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch 2 times, most recently from f550411 to e80a19b Compare April 22, 2025 21:31
Copy link
Contributor

@bantonsson bantonsson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good with cargo tree -p ddcommon --features fips

@apiarian-datadog apiarian-datadog force-pushed the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch from f874750 to 404e0eb Compare April 23, 2025 16:21
@codecov-commenter
Copy link

codecov-commenter commented Apr 23, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.38%. Comparing base (d433dd2) to head (1a60690).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1028      +/-   ##
==========================================
+ Coverage   71.35%   71.38%   +0.03%     
==========================================
  Files         329      329              
  Lines       49165    49182      +17     
==========================================
+ Hits        35082    35110      +28     
+ Misses      14083    14072      -11     
Components Coverage Δ
crashtracker 42.84% <ø> (ø)
crashtracker-ffi 6.30% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 90.78% <ø> (-0.15%) ⬇️
data-pipeline-ffi 89.02% <ø> (ø)
ddcommon 78.29% <ø> (ø)
ddcommon-ffi 66.37% <ø> (ø)
ddtelemetry 60.19% <ø> (ø)
ddtelemetry-ffi 21.32% <ø> (ø)
dogstatsd-client 82.57% <ø> (ø)
ipc 82.54% <ø> (ø)
profiling 77.49% <ø> (ø)
profiling-ffi 62.12% <ø> (ø)
sidecar 42.85% <ø> (+0.26%) ⬆️
sidecar-ffi 15.32% <ø> (+2.04%) ⬆️
spawn-worker 54.37% <ø> (ø)
tinybytes 90.54% <ø> (ø)
trace-normalization 98.24% <ø> (ø)
trace-obfuscation 94.16% <ø> (ø)
trace-protobuf 78.50% <ø> (ø)
trace-utils 89.33% <ø> (-0.14%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@apiarian-datadog apiarian-datadog force-pushed the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch from 097862c to eef7b6e Compare April 23, 2025 19:20
Copy link
Contributor

@morrisonlevi morrisonlevi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, the crypto provider and certificates stuff is a mess. We're shipping ring and aws-lc-rs, and rust-native-certs and webpki.

But I won't hold status quo against you. Just know in the future if we straighten it out, you may need to adjust features and code to match.

@apiarian-datadog
Copy link
Contributor Author

Honestly, the crypto provider and certificates stuff is a mess. We're shipping ring and aws-lc-rs, and rust-native-certs and webpki.

But I won't hold status quo against you. Just know in the future if we straighten it out, you may need to adjust features and code to match.

ya, this was pretty tricky to navigate. we have a build script in bottlecap which checks that ring (and a few other non-fips-ish dependencies) aren't included at compile time, so we should hopefully be able to catch it when these things get updated in libdatadog

fips = ["https", "hyper-rustls/fips"]

[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(coverage)'] }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like a comment here to explain what this is doing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

/// happens here. On non-unix platforms, ddcommon uses `ring` instead, which handles this
/// at rustls initialization. TODO: Move to the more ergonomic LazyLock when MSRV is 1.80
/// In fips mode we expect someone to have done this already.
#[cfg(any(not(feature = "fips"), coverage))]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg(any(not(feature = "fips"), coverage))]
#[cfg(any(not(feature = "fips"), coverage))]

Can't there be a default here? instead of doing not feature?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would we write that in a cfg line? we want this to be run in both coverage checks or for a non-fips build

@apiarian-datadog apiarian-datadog force-pushed the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch from d3bc0f3 to 6459250 Compare April 24, 2025 15:41
@apiarian-datadog apiarian-datadog force-pushed the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch from 6459250 to 4c441b2 Compare April 24, 2025 16:33
@apiarian-datadog apiarian-datadog enabled auto-merge (squash) April 24, 2025 16:35
@apiarian-datadog apiarian-datadog force-pushed the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch from 4c441b2 to 9132f4e Compare April 24, 2025 16:43
@apiarian-datadog apiarian-datadog force-pushed the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch from 9132f4e to 1a60690 Compare April 24, 2025 16:46
@apiarian-datadog apiarian-datadog merged commit d6a2da3 into main Apr 24, 2025
35 checks passed
@apiarian-datadog apiarian-datadog deleted the aleksandr.pasechnik/svls-6242-fips-features-for-bottlecap branch April 24, 2025 17:14
danielsn pushed a commit that referenced this pull request May 22, 2025
We need a `ddcommon/fips` and a `trace_utils/fips` feature. This functionality is written specifically for use in the datadog-lambda-extension. Other use cases should probably validate carefully that the right dependencies are being used in their builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants