Skip to content

test(prof)!: fix fuzzer, stop sorting label ids #1026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 16, 2025
Merged

test(prof)!: fix fuzzer, stop sorting label ids #1026

merged 2 commits into from
Apr 16, 2025

Conversation

@morrisonlevi
Copy link
Contributor

@morrisonlevi morrisonlevi commented Apr 15, 2025 •
edited
Loading

PROF-11530

What does this PR do?

Fixes #958 tests by:

  1. Generating unique Label.key from the start (rather than checking later that they are unique, and failing).
  2. LabelSet no longer sorts LabelIds so that the input to output order to remains the same.

This also:

  1. Removes pub from the fuzzer items, since nothing outside the modules uses them (the module isn't public, either). This allows lints like warn(dead_code) to work.

This PR is technically a breaking change; see additional notes.

Motivation

Fuzzer failures were found in CI in fuzz_add_sample_with_fixed_sample_length.

Additional Notes

Since LabelSet no longer sorts the LabelIds before deduplicating the whole set, you need to be sure that your profiler always adds labels in the same order. The order doesn't matter, what matters is that it's consistent.

How to test the change?

Update your code to add labels in the same order every time, and then test as usual (no other changes).

@github-actions github-actions bot added the profiling Relates to the profiling* modules. label Apr 15, 2025
@pr-commenter
Copy link

pr-commenter bot commented Apr 15, 2025 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2025-04-16 03:36:20

Comparing candidate commit b82f0a6 in PR branch levi/labels with baseline commit 6596921 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.340µs 2.391µs ± 0.017µs 2.389µs ± 0.007µs 2.400µs 2.428µs 2.433µs 2.436µs 1.98% -0.197 1.929 0.72% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.389µs; 2.394µs] or [-0.100%; +0.100%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 5.963ms 5.976ms ± 0.007ms 5.975ms ± 0.004ms 5.979ms 5.986ms 5.993ms 6.024ms 0.82% 2.190 11.628 0.12% 0.000ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [5.975ms; 5.977ms] or [-0.016%; +0.016%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 67.029µs 67.263µs ± 0.216µs 67.248µs ± 0.061µs 67.302µs 67.467µs 67.610µs 69.949µs 4.02% 9.709 117.598 0.32% 0.015µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [67.233µs; 67.293µs] or [-0.045%; +0.045%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 3.895µs 3.912µs ± 0.003µs 3.912µs ± 0.001µs 3.914µs 3.917µs 3.919µs 3.921µs 0.22% -0.960 8.258 0.07% 0.000µs 1 200
credit_card/is_card_number/ throughput 255048249.703op/s 255593467.917op/s ± 176835.618op/s 255597134.940op/s ± 82818.688op/s 255667894.514op/s 255868822.249op/s 255978823.399op/s 256728119.099op/s 0.44% 0.979 8.367 0.07% 12504.166op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 76.819µs 77.591µs ± 0.417µs 77.527µs ± 0.234µs 77.820µs 78.335µs 78.749µs 79.183µs 2.14% 0.821 0.793 0.54% 0.029µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 12628925.173op/s 12888381.340op/s ± 68968.882op/s 12898770.106op/s ± 38990.040op/s 12933789.540op/s 12983920.762op/s 13015529.997op/s 13017572.108op/s 0.92% -0.788 0.709 0.53% 4876.836op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 70.923µs 71.375µs ± 0.271µs 71.317µs ± 0.176µs 71.535µs 71.869µs 72.121µs 72.451µs 1.59% 0.814 0.814 0.38% 0.019µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 13802448.202op/s 14010710.180op/s ± 52975.572op/s 14021953.805op/s ± 34654.149op/s 14051260.043op/s 14080880.559op/s 14098712.308op/s 14099702.496op/s 0.55% -0.790 0.742 0.38% 3745.939op/s 1 200
credit_card/is_card_number/37828224631 execution_time 3.892µs 3.912µs ± 0.003µs 3.912µs ± 0.001µs 3.914µs 3.917µs 3.918µs 3.919µs 0.17% -1.632 8.667 0.08% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 255181578.765op/s 255610029.184op/s ± 196920.265op/s 255604818.151op/s ± 81066.738op/s 255666812.244op/s 255975017.483op/s 256061350.601op/s 256914735.467op/s 0.51% 1.651 8.803 0.08% 13924.365op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 67.143µs 68.040µs ± 0.382µs 68.065µs ± 0.279µs 68.290µs 68.635µs 68.933µs 69.097µs 1.52% 0.061 -0.281 0.56% 0.027µs 1 200
credit_card/is_card_number/378282246310005 throughput 14472361.334op/s 14697665.416op/s ± 82584.703op/s 14691884.981op/s ± 60387.483op/s 14759914.962op/s 14824563.401op/s 14891702.464op/s 14893595.896op/s 1.37% -0.032 -0.293 0.56% 5839.620op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 51.732µs 51.836µs ± 0.038µs 51.832µs ± 0.017µs 51.852µs 51.901µs 51.960µs 51.998µs 0.32% 0.749 2.751 0.07% 0.003µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 19231639.215op/s 19291801.438op/s ± 14073.580op/s 19293232.899op/s ± 6400.019op/s 19298680.453op/s 19311645.436op/s 19324832.468op/s 19330410.602op/s 0.19% -0.740 2.728 0.07% 995.152op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 6.027µs 6.054µs ± 0.024µs 6.040µs ± 0.004µs 6.074µs 6.111µs 6.115µs 6.119µs 1.32% 1.104 0.163 0.40% 0.002µs 1 200
credit_card/is_card_number/x371413321323331 throughput 163419434.515op/s 165176679.780op/s ± 654186.062op/s 165575633.792op/s ± 121426.090op/s 165658858.503op/s 165713675.193op/s 165906820.132op/s 165926571.921op/s 0.21% -1.093 0.128 0.40% 46257.940op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 3.896µs 3.913µs ± 0.003µs 3.913µs ± 0.001µs 3.914µs 3.917µs 3.919µs 3.920µs 0.18% -1.109 6.746 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 255096846.979op/s 255559222.598op/s ± 181769.391op/s 255562061.812op/s ± 91472.038op/s 255653431.567op/s 255888301.808op/s 255990358.162op/s 256679645.927op/s 0.44% 1.125 6.839 0.07% 12853.037op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 66.022µs 66.475µs ± 0.238µs 66.467µs ± 0.146µs 66.605µs 66.891µs 67.104µs 67.594µs 1.70% 0.974 2.166 0.36% 0.017µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 14794306.720op/s 15043466.185op/s ± 53769.242op/s 15045126.664op/s ± 32965.814op/s 15084237.734op/s 15118483.935op/s 15134413.486op/s 15146378.210op/s 0.67% -0.940 2.026 0.36% 3802.060op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 59.534µs 59.684µs ± 0.067µs 59.679µs ± 0.044µs 59.724µs 59.793µs 59.869µs 59.961µs 0.47% 0.707 1.180 0.11% 0.005µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 16677507.046op/s 16754873.147op/s ± 18860.057op/s 16756323.804op/s ± 12214.404op/s 16767774.435op/s 16780839.218op/s 16789670.520op/s 16797141.127op/s 0.24% -0.698 1.154 0.11% 1333.607op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 3.897µs 3.913µs ± 0.003µs 3.913µs ± 0.001µs 3.914µs 3.917µs 3.918µs 3.921µs 0.21% -1.019 5.150 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 255034486.530op/s 255573383.422op/s ± 180762.659op/s 255562905.050op/s ± 85192.359op/s 255649570.452op/s 255904633.995op/s 256011333.800op/s 256608394.006op/s 0.41% 1.032 5.212 0.07% 12781.850op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 56.222µs 56.444µs ± 0.103µs 56.438µs ± 0.065µs 56.501µs 56.635µs 56.717µs 56.742µs 0.54% 0.457 0.184 0.18% 0.007µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 17623598.355op/s 17716880.433op/s ± 32436.635op/s 17718538.136op/s ± 20537.901op/s 17740519.586op/s 17763232.430op/s 17782954.665op/s 17786569.361op/s 0.38% -0.446 0.171 0.18% 2293.616op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 51.738µs 51.829µs ± 0.036µs 51.828µs ± 0.020µs 51.851µs 51.887µs 51.926µs 51.946µs 0.23% 0.012 0.789 0.07% 0.003µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 19250579.911op/s 19294270.938op/s ± 13513.854op/s 19294611.016op/s ± 7379.020op/s 19300884.946op/s 19318394.397op/s 19327350.714op/s 19328194.037op/s 0.17% -0.006 0.786 0.07% 955.574op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 6.025µs 6.052µs ± 0.025µs 6.039µs ± 0.006µs 6.071µs 6.102µs 6.124µs 6.186µs 2.44% 1.732 4.222 0.42% 0.002µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 161642499.011op/s 165233146.011op/s ± 682753.128op/s 165584731.225op/s ± 154369.720op/s 165684346.273op/s 165876536.250op/s 165950235.178op/s 165966238.270op/s 0.23% -1.692 3.962 0.41% 48277.937op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [3.912µs; 3.913µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/ throughput [255568960.201op/s; 255617975.633op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [77.534µs; 77.649µs] or [-0.074%; +0.074%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [12878822.917op/s; 12897939.764op/s] or [-0.074%; +0.074%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [71.337µs; 71.413µs] or [-0.053%; +0.053%] None None None
credit_card/is_card_number/ 378282246310005 throughput [14003368.276op/s; 14018052.085op/s] or [-0.052%; +0.052%] None None None
credit_card/is_card_number/37828224631 execution_time [3.912µs; 3.913µs] or [-0.011%; +0.011%] None None None
credit_card/is_card_number/37828224631 throughput [255582737.929op/s; 255637320.439op/s] or [-0.011%; +0.011%] None None None
credit_card/is_card_number/378282246310005 execution_time [67.987µs; 68.093µs] or [-0.078%; +0.078%] None None None
credit_card/is_card_number/378282246310005 throughput [14686219.971op/s; 14709110.862op/s] or [-0.078%; +0.078%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [51.830µs; 51.841µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [19289850.976op/s; 19293751.901op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/x371413321323331 execution_time [6.051µs; 6.058µs] or [-0.055%; +0.055%] None None None
credit_card/is_card_number/x371413321323331 throughput [165086015.884op/s; 165267343.677op/s] or [-0.055%; +0.055%] None None None
credit_card/is_card_number_no_luhn/ execution_time [3.913µs; 3.913µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ throughput [255534031.109op/s; 255584414.088op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [66.442µs; 66.508µs] or [-0.050%; +0.050%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [15036014.285op/s; 15050918.085op/s] or [-0.050%; +0.050%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [59.675µs; 59.694µs] or [-0.016%; +0.016%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [16752259.325op/s; 16757486.970op/s] or [-0.016%; +0.016%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [3.912µs; 3.913µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [255548331.456op/s; 255598435.388op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [56.429µs; 56.458µs] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [17712385.028op/s; 17721375.839op/s] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [51.824µs; 51.834µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [19292398.048op/s; 19296143.828op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [6.049µs; 6.056µs] or [-0.058%; +0.058%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [165138522.994op/s; 165327769.028op/s] or [-0.057%; +0.057%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 208.818µs 209.376µs ± 0.255µs 209.367µs ± 0.120µs 209.483µs 209.672µs 209.760µs 211.512µs 1.02% 3.624 27.197 0.12% 0.018µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 4727870.349op/s 4776094.502op/s ± 5797.086op/s 4776295.880op/s ± 2735.324op/s 4779061.022op/s 4783320.529op/s 4786720.794op/s 4788868.116op/s 0.26% -3.566 26.620 0.12% 409.916op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 18.601µs 18.687µs ± 0.032µs 18.685µs ± 0.019µs 18.710µs 18.738µs 18.755µs 18.772µs 0.47% -0.207 0.269 0.17% 0.002µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 53269578.376op/s 53511886.118op/s ± 90406.087op/s 53518974.635op/s ± 53280.262op/s 53563772.555op/s 53658480.687op/s 53760012.888op/s 53761358.796op/s 0.45% 0.218 0.278 0.17% 6392.676op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 10.848µs 10.947µs ± 0.035µs 10.946µs ± 0.024µs 10.971µs 11.010µs 11.030µs 11.043µs 0.89% 0.049 0.258 0.32% 0.002µs 1 200
normalization/normalize_name/normalize_name/good throughput 90553273.536op/s 91349968.461op/s ± 290017.091op/s 91360526.918op/s ± 201703.166op/s 91513721.481op/s 91858237.021op/s 91969488.024op/s 92179622.631op/s 0.90% -0.028 0.258 0.32% 20507.305op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [209.341µs; 209.412µs] or [-0.017%; +0.017%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [4775291.082op/s; 4776897.923op/s] or [-0.017%; +0.017%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [18.683µs; 18.692µs] or [-0.023%; +0.023%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [53499356.704op/s; 53524415.532op/s] or [-0.023%; +0.023%] None None None
normalization/normalize_name/normalize_name/good execution_time [10.942µs; 10.952µs] or [-0.044%; +0.044%] None None None
normalization/normalize_name/normalize_name/good throughput [91309774.882op/s; 91390162.041op/s] or [-0.044%; +0.044%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 18.177µs 26.595µs ± 10.955µs 18.506µs ± 0.238µs 36.315µs 45.108µs 52.357µs 78.671µs 325.10% 1.289 2.185 41.09% 0.775µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [25.077µs; 28.113µs] or [-5.709%; +5.709%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 71.645ms 71.987ms ± 0.274ms 71.949ms ± 0.147ms 72.079ms 72.573ms 72.722ms 73.379ms 1.99% 1.588 3.350 0.38% 0.019ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [71.949ms; 72.025ms] or [-0.053%; +0.053%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 33.315µs 34.160µs ± 1.140µs 33.460µs ± 0.063µs 35.531µs 36.357µs 36.413µs 37.361µs 11.66% 1.002 -0.794 3.33% 0.081µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [34.002µs; 34.318µs] or [-0.462%; +0.462%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
ip_address/quantize_peer_ip_address_benchmark execution_time 4.989µs 5.058µs ± 0.042µs 5.050µs ± 0.034µs 5.100µs 5.125µs 5.128µs 5.132µs 1.62% 0.221 -1.358 0.84% 0.003µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark execution_time [5.052µs; 5.064µs] or [-0.116%; +0.116%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 503.994µs 505.249µs ± 0.580µs 505.254µs ± 0.338µs 505.570µs 506.166µs 506.554µs 508.974µs 0.74% 1.391 7.415 0.11% 0.041µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1964738.832op/s 1979225.558op/s ± 2270.102op/s 1979201.669op/s ± 1322.490op/s 1980633.912op/s 1982532.097op/s 1983541.378op/s 1984150.605op/s 0.25% -1.365 7.234 0.11% 160.520op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 453.959µs 455.773µs ± 0.540µs 455.702µs ± 0.320µs 456.085µs 456.695µs 457.135µs 457.407µs 0.37% 0.064 0.666 0.12% 0.038µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2186235.092op/s 2194076.100op/s ± 2600.959op/s 2194415.323op/s ± 1538.904op/s 2195477.646op/s 2198251.209op/s 2200276.345op/s 2202840.409op/s 0.38% -0.054 0.669 0.12% 183.916op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 176.969µs 177.668µs ± 0.247µs 177.723µs ± 0.168µs 177.831µs 178.032µs 178.245µs 178.437µs 0.40% -0.058 0.018 0.14% 0.017µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5604216.558op/s 5628485.729op/s ± 7813.140op/s 5626720.996op/s ± 5307.151op/s 5634537.380op/s 5641635.330op/s 5645003.853op/s 5650699.398op/s 0.43% 0.067 0.013 0.14% 552.472op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 37.556µs 37.650µs ± 0.042µs 37.646µs ± 0.027µs 37.674µs 37.725µs 37.773µs 37.785µs 0.37% 0.606 0.415 0.11% 0.003µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 26465635.494op/s 26560602.030op/s ± 29876.267op/s 26563075.674op/s ± 19286.257op/s 26582158.431op/s 26604781.841op/s 26613484.848op/s 26626851.751op/s 0.24% -0.600 0.403 0.11% 2112.571op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 48.088µs 48.284µs ± 0.246µs 48.102µs ± 0.011µs 48.520µs 48.641µs 48.694µs 49.868µs 3.67% 1.673 6.642 0.51% 0.017µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 20052877.896op/s 20711256.509op/s ± 104583.758op/s 20789168.250op/s ± 4688.651op/s 20791659.574op/s 20793708.893op/s 20794493.761op/s 20795303.826op/s 0.03% -1.588 5.889 0.50% 7395.188op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [505.168µs; 505.329µs] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1978910.943op/s; 1979540.172op/s] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [455.698µs; 455.848µs] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2193715.632op/s; 2194436.568op/s] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [177.634µs; 177.702µs] or [-0.019%; +0.019%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5627402.903op/s; 5629568.555op/s] or [-0.019%; +0.019%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [37.644µs; 37.656µs] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [26556461.467op/s; 26564742.593op/s] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [48.250µs; 48.318µs] or [-0.071%; +0.071%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [20696762.206op/s; 20725750.812op/s] or [-0.070%; +0.070%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 245.845ns 255.057ns ± 11.890ns 249.201ns ± 1.468ns 259.166ns 283.668ns 291.221ns 294.924ns 18.35% 1.748 2.029 4.65% 0.841ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [253.410ns; 256.705ns] or [-0.646%; +0.646%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 149.060µs 149.937µs ± 0.428µs 149.864µs ± 0.139µs 150.046µs 150.561µs 151.188µs 153.311µs 2.30% 3.828 24.376 0.28% 0.030µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [149.878µs; 149.996µs] or [-0.040%; +0.040%] None None None

Group 13

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz b82f0a6 1744759839 levi/labels
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.185µs 3.176µs ± 1.432µs 2.990µs ± 0.031µs 3.014µs 3.636µs 14.063µs 14.872µs 397.40% 7.404 55.688 44.97% 0.101µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.977µs; 3.374µs] or [-6.248%; +6.248%] None None None

Baseline

Omitted due to size.

@r1viollet
Copy link
Contributor

r1viollet commented Apr 15, 2025 •
edited
Loading

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 78.03 MB 78.00 MB --.03% (-31.60 KB) 💪
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 7.79 MB 7.78 MB -0% (-592 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so.debug 24.21 MB 24.20 MB --.03% (-8.14 KB) 💪
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 7.71 MB 7.71 MB -0% (-592 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 72.40 MB 72.36 MB --.04% (-32.82 KB) 💪
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so.debug 22.81 MB 22.81 MB --.03% (-8.58 KB) 💪
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 17.01 MB 17.00 MB --.08% (-14.50 KB) 💪
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 61.83 KB 61.83 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 114.18 MB 114.14 MB --.03% (-40.00 KB) 💪
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 633.63 MB 633.59 MB -0% (-38.72 KB) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 5.06 MB 5.06 MB --.03% (-2.00 KB) 💪
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 61.83 KB 61.83 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 16.15 MB 16.14 MB --.04% (-8.00 KB) 💪
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 26.83 MB 26.82 MB --.05% (-15.80 KB) 💪
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 14.40 MB 14.38 MB --.09% (-13.50 KB) 💪
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 62.78 KB 62.78 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 116.19 MB 116.13 MB --.04% (-56.00 KB) 💪
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 625.37 MB 625.31 MB --.01% (-67.79 KB) 💪
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 3.83 MB 3.83 MB +.01% (+512 B) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 62.78 KB 62.78 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 16.77 MB 16.76 MB --.04% (-8.00 KB) 💪
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 24.74 MB 24.72 MB --.06% (-17.05 KB) 💪
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 67.28 MB 67.25 MB --.03% (-24.40 KB) 💪
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.27 MB 8.27 MB -0% (-312 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so.debug 23.34 MB 23.34 MB --.02% (-6.19 KB) 💪
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 68.24 MB 68.22 MB --.03% (-24.07 KB) 💪
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 8.15 MB 8.14 MB --.05% (-4.31 KB) 💪
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so.debug 20.99 MB 20.98 MB --.02% (-6.25 KB) 💪

@codecov-commenter
Copy link

codecov-commenter commented Apr 15, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 77.70701% with 35 lines in your changes missing coverage. Please review.

Project coverage is 71.52%. Comparing base (6596921) to head (b82f0a6).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1026      +/-   ##
==========================================
- Coverage   71.56%   71.52%   -0.04%     
==========================================
  Files         339      339              
  Lines       50683    50751      +68     
==========================================
+ Hits        36270    36300      +30     
- Misses      14413    14451      +38
Components Coverage Δ
crashtracker 42.79% <ø> (-0.03%) ⬇️
crashtracker-ffi 6.30% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 90.93% <ø> (ø)
data-pipeline-ffi 90.35% <ø> (ø)
ddcommon 78.57% <ø> (ø)
ddcommon-ffi 66.37% <ø> (ø)
ddtelemetry 60.29% <ø> (ø)
ddtelemetry-ffi 21.43% <ø> (ø)
dogstatsd-client 82.57% <ø> (ø)
ipc 82.41% <ø> (-0.11%) ⬇️
profiling 77.49% <77.70%> (-0.05%) ⬇️
profiling-ffi 62.12% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 41.14% <ø> (ø)
sidecar-ffi 2.05% <ø> (ø)
spawn-worker 54.37% <ø> (ø)
tinybytes 89.86% <ø> (ø)
trace-mini-agent 73.82% <ø> (ø)
trace-normalization 98.24% <ø> (ø)
trace-obfuscation 96.00% <ø> (ø)
trace-protobuf 78.50% <ø> (ø)
trace-utils 92.82% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@morrisonlevi morrisonlevi marked this pull request as ready for review April 16, 2025 03:22
@morrisonlevi morrisonlevi requested review from a team as code owners April 16, 2025 03:22
@github-actions github-actions bot removed the ci-build label Apr 16, 2025
@morrisonlevi
Copy link
Contributor Author

A windows job and also the buildx bake job are failing regularly enough I can't get this CI green.

I'm going to merge this, I'd still like a review.

@morrisonlevi morrisonlevi merged commit daf50ad into main Apr 16, 2025
63 of 68 checks passed
@morrisonlevi morrisonlevi deleted the levi/labels branch April 16, 2025 03:41
@morrisonlevi morrisonlevi mentioned this pull request Apr 19, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

profiling Relates to the profiling* modules.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] Fuzz tests failure in profiling with labels

4 participants

@morrisonlevi @r1viollet @codecov-commenter