Skip to content

perf: calculate v0.6.7 baseline #367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 13 commits into
base: main
Choose a base branch
from
Draft

perf: calculate v0.6.7 baseline #367

wants to merge 13 commits into from
+714 −704

Conversation

maksymar
Copy link
Contributor

@maksymar maksymar commented Jul 7, 2025

No description provided.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 7, 2025 •
edited
Loading

canbench 🏋 (dir: ./benchmarks/memory_manager) 5663bf8 2025-07-18 05:29:49 UTC

✅ ./benchmarks/memory_manager/canbench_results.yml is up to date
📦 canbench_results_memory-manager.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max +3.17M | p75 +1.66M | median +153.92K | p25 +76.96K | min 0]
    change %: [max +0.91% | p75 +0.46% | median +0.01% | p25 0.01% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 8, 2025 •
edited
Loading

canbench 🏋 (dir: ./benchmarks/vec) 5663bf8 2025-07-18 05:29:51 UTC

✅ ./benchmarks/vec/canbench_results.yml is up to date
📦 canbench_results_vec.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   Regressions detected 🔴
    counts:   [total 16 | regressed 4 | improved 0 | new 0 | unchanged 12]
    change:   [max +8.84M | p75 +120.00K | median -20.00K | p25 -20.00K | min -102.33K]
    change %: [max +101.83% | p75 +2.28% | median -0.51% | p25 -0.61% | min -1.79%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                        | calls |    ins |   ins Δ% | HI |  HI Δ% | SMI |  SMI Δ% |
|--------|-----------------------------|-------|--------|----------|----|--------|-----|---------|
|   +    | vec_get_blob_4_mem_manager  |       | 14.51M | +101.83% |  0 |  0.00% |   0 |   0.00% |
|   +    | vec_get_blob_64_mem_manager |       | 22.49M |  +64.73% |  0 |  0.00% |   0 |   0.00% |
|   +    | vec_get_u64                 |       |  5.27M |  +10.02% |  0 |  0.00% |   0 |   0.00% |
|   +    | vec_insert_u64              |       |  5.87M |   +9.11% |  0 |  0.00% |   1 |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 8, 2025 •
edited
Loading

canbench 🏋 (dir: ./benchmarks/compare) ce30e2f 2025-07-08 17:16:25 UTC

✅ ./benchmarks/compare/canbench_results.yml is up to date
📦 canbench_results_compare.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   Regressions detected 🔴
    counts:   [total 18 | regressed 12 | improved 0 | new 0 | unchanged 6]
    change:   [max +85.19B | p75 +856.87M | median +288.62M | p25 +361.15K | min +986]
    change %: [max +517.21% | p75 +158.04% | median +97.42% | p25 +0.19% | min 0.00%]

  heap_increase:
    status:   Regressions and improvements 🔴🟢
    counts:   [total 18 | regressed 2 | improved 1 | new 0 | unchanged 15]
    change:   [max +1.62K | p75 0 | median 0 | p25 0 | min -1.60K]
    change %: [max +12453.85% | p75 0.00% | median 0.00% | p25 0.00% | min -100.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 18 | regressed 0 | improved 0 | new 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                     | calls |     ins |   ins Δ% |    HI |      HI Δ% |   SMI |  SMI Δ% |
|--------|--------------------------|-------|---------|----------|-------|------------|-------|---------|
|  +/-   | read_chunks_btreemap_1   |       | 917.93M | +517.21% |     0 |   -100.00% |     0 |   0.00% |
|   +    | read_chunks_vec_1        |       | 393.22M | +275.00% |     0 |      0.00% |     0 |   0.00% |
|   +    | read_chunks_vec_1k       |       | 394.72M | +272.98% |     0 |      0.00% |     0 |   0.00% |
|   +    | read_chunks_btreemap_1k  |       |   1.72B | +244.73% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_btreemap_1  |       | 957.54M | +168.07% | 1.63K | +12453.85% | 1.54K |   0.00% |
|   +    | write_chunks_btreemap_1k |       |   9.55B | +127.97% |     4 |   +100.00% | 1.54K |   0.00% |
|   +    | read_chunks_btreemap_1m  |       |  87.47B | +113.66% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_btreemap_1m |       | 168.85B | +101.83% |     0 |      0.00% | 3.07K |   0.00% |
|   +    | read_chunks_stable_1m    |       | 462.00M | +100.87% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_stable_1m   |       | 495.41M |  +93.97% |     0 |      0.00% | 1.66K |   0.00% |
|   +    | read_chunks_vec_1m       |       |   1.90B |  +87.65% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_vec_1m      |       |   2.64B |  +39.28% |     0 |      0.00% | 1.54K |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 8, 2025 •
edited
Loading

canbench 🏋 (dir: ./benchmarks/btreemap) 5663bf8 2025-07-18 05:31:52 UTC

✅ ./benchmarks/btreemap/canbench_results.yml is up to date
📦 canbench_results_btreemap.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   Regressions and improvements 🔴🟢
    counts:   [total 303 | regressed 204 | improved 23 | new 0 | unchanged 76]
    change:   [max +1.77B | p75 +132.45M | median +26.90M | p25 +2.11M | min -883.02M]
    change %: [max +5890.51% | p75 +15.76% | median +5.44% | p25 +0.70% | min -11.99%]

  heap_increase:
    status:   Regressions detected 🔴
    counts:   [total 303 | regressed 1 | improved 0 | new 0 | unchanged 302]
    change:   [max +161 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max +100.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 303 | regressed 0 | improved 0 | new 0 | unchanged 303]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                                         | calls |     ins |    ins Δ% | HI |  HI Δ% | SMI |  SMI Δ% |
|--------|----------------------------------------------|-------|---------|-----------|----|--------|-----|---------|
|   +    | btreemap_v2_range_key_sum_20_10mib           |       |   1.11B | +5890.51% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_scan_iter_rev_20_10mib           |       |   1.10B | +5880.02% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_scan_iter_20_10mib               |       |   1.10B | +5880.02% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_scan_iter_rev_1k_10kib           |       |  57.08M | +2205.78% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_scan_iter_1k_10kib               |       |  57.10M | +2191.72% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_range_key_sum_1k_10kib           |       |  57.29M | +2126.55% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_contains_10mib_values            |       |   1.23B |  +766.50% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_get_10mib_values                 |       |   1.23B |  +217.11% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_range_value_sum_20_10mib         |       |   1.11B |  +177.79% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_range_value_sum_1k_10kib         |       |  57.30M |  +177.24% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_contains_u64_blob512 |       | 658.72M |  +136.22% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_contains_u64_u64     |       | 652.60M |  +130.17% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_contains_vec512_u64  |       |   2.72B |  +126.28% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_get_u64_u64          |       | 652.65M |  +123.72% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_get_u64_blob512      |       | 658.77M |  +123.01% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_get_vec512_u64       |       |   2.72B |  +118.80% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_contains_u64_vec512  |       | 795.11M |  +116.63% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_get_u64_vec512       |       | 795.16M |  +104.50% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_remove_u64_u64       |       |   1.43B |   +94.35% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_insert_u64_u64       |       |   1.01B |   +94.35% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_insert_u64_blob512   |       |   1.17B |   +91.84% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_remove_u64_blob512   |       |   1.69B |   +91.09% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_insert_u64_vec512    |       |   1.56B |   +86.65% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_remove_u64_vec512    |       |   2.25B |   +83.51% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_insert_vec512_u64    |       |   3.33B |   +69.52% |  0 |  0.00% |   0 |   0.00% |
|  ...   | ... 177 rows omitted ...                     |       |         |           |    |        |     |         |
|   +    | btreemap_v2_pop_first_blob_32_512            |       | 978.42M |    +2.18% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_get_blob_32_0                    |       | 344.27M |    +2.03% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_insert_blob_32_8                 |       | 507.19M |    -2.01% |  0 |  0.00% |   9 |   0.00% |
|   -    | btreemap_v2_scan_keys_rev_1k_0b              |       | 963.72K |    -2.08% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_get_vec_16_128                   |       | 440.23M |    -2.11% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_insert_principal                 |       | 492.47M |    -2.15% |  0 |  0.00% |   8 |   0.00% |
|   -    | btreemap_v2_insert_blob_32_32                |       | 517.50M |    -2.17% |  0 |  0.00% |  13 |   0.00% |
|   -    | btreemap_v2_insert_blob_32_64                |       | 522.98M |    -2.23% |  0 |  0.00% |  18 |   0.00% |
|   -    | btreemap_v2_remove_vec_32_32                 |       | 847.18M |    -2.35% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_remove_vec_32_0                  |       | 813.72M |    -2.47% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_remove_blob_32_0                 |       | 635.11M |    -2.93% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_insert_blob_32_0                 |       | 473.38M |    -3.48% |  0 |  0.00% |   8 |   0.00% |
|   -    | btreemap_v2_remove_vec_32_16                 |       | 840.65M |    -3.52% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_scan_keys_1k_0b                  |       | 946.48K |    -3.68% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_insert_blob_128_128              |       |   1.14B |    -3.70% |  0 |  0.00% |  46 |   0.00% |
|   -    | btreemap_v2_scan_keys_rev_1k_10kib           |       |   2.39M |    -3.71% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_remove_vec_32_64                 |       | 929.99M |    -3.96% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_scan_keys_1k_10kib               |       |   2.39M |    -4.19% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_insert_blob_256_128              |       |   1.69B |    -5.68% |  0 |  0.00% |  67 |   0.00% |
|   -    | btreemap_v2_insert_blob_512_128              |       |   2.86B |    -6.01% |  0 |  0.00% | 111 |   0.00% |
|   -    | btreemap_v2_insert_blob_1024_128             |       |   5.11B |    -7.07% |  0 |  0.00% | 196 |   0.00% |
|   -    | btreemap_v2_remove_blob_128_128              |       |   1.47B |    -7.30% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_remove_blob_256_128              |       |   2.20B |    -9.08% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_remove_blob_512_128              |       |   3.61B |   -11.41% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_remove_blob_1024_128             |       |   6.48B |   -11.99% |  0 |  0.00% |   0 |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 10, 2025 •
edited
Loading

canbench 🏋 (dir: ./benchmarks/io_chunks) 5663bf8 2025-07-18 05:31:13 UTC

✅ ./benchmarks/io_chunks/canbench_results.yml is up to date
📦 canbench_results_io_chunks.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   Regressions detected 🔴
    counts:   [total 18 | regressed 12 | improved 0 | new 0 | unchanged 6]
    change:   [max +85.19B | p75 +856.87M | median +288.62M | p25 +361.15K | min +986]
    change %: [max +517.21% | p75 +158.04% | median +97.42% | p25 +0.19% | min 0.00%]

  heap_increase:
    status:   Regressions and improvements 🔴🟢
    counts:   [total 18 | regressed 2 | improved 1 | new 0 | unchanged 15]
    change:   [max +1.62K | p75 0 | median 0 | p25 0 | min -1.60K]
    change %: [max +12453.85% | p75 0.00% | median 0.00% | p25 0.00% | min -100.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 18 | regressed 0 | improved 0 | new 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                     | calls |     ins |   ins Δ% |    HI |      HI Δ% |   SMI |  SMI Δ% |
|--------|--------------------------|-------|---------|----------|-------|------------|-------|---------|
|  +/-   | read_chunks_btreemap_1   |       | 917.93M | +517.21% |     0 |   -100.00% |     0 |   0.00% |
|   +    | read_chunks_vec_1        |       | 393.22M | +275.00% |     0 |      0.00% |     0 |   0.00% |
|   +    | read_chunks_vec_1k       |       | 394.72M | +272.98% |     0 |      0.00% |     0 |   0.00% |
|   +    | read_chunks_btreemap_1k  |       |   1.72B | +244.73% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_btreemap_1  |       | 957.54M | +168.07% | 1.63K | +12453.85% | 1.54K |   0.00% |
|   +    | write_chunks_btreemap_1k |       |   9.55B | +127.97% |     4 |   +100.00% | 1.54K |   0.00% |
|   +    | read_chunks_btreemap_1m  |       |  87.47B | +113.66% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_btreemap_1m |       | 168.85B | +101.83% |     0 |      0.00% | 3.07K |   0.00% |
|   +    | read_chunks_stable_1m    |       | 462.00M | +100.87% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_stable_1m   |       | 495.41M |  +93.97% |     0 |      0.00% | 1.66K |   0.00% |
|   +    | read_chunks_vec_1m       |       |   1.90B |  +87.65% |     0 |      0.00% |     0 |   0.00% |
|   +    | write_chunks_vec_1m      |       |   2.64B |  +39.28% |     0 |      0.00% | 1.54K |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 14, 2025 •
edited
Loading

canbench 🏋 (dir: ./benchmarks/nns) 5663bf8 2025-07-18 05:30:11 UTC

✅ ./benchmarks/nns/canbench_results.yml is up to date
📦 canbench_results_nns.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   Regressions detected 🔴
    counts:   [total 16 | regressed 8 | improved 0 | new 0 | unchanged 8]
    change:   [max +7.96B | p75 +355.99M | median +33.92K | p25 +38 | min +38]
    change %: [max +110.78% | p75 +100.78% | median +38.10% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                                  | calls |     ins |   ins Δ% | HI |  HI Δ% | SMI |  SMI Δ% |
|--------|---------------------------------------|-------|---------|----------|----|--------|-----|---------|
|   +    | vote_cascading_stable_centralized_10k |       |   2.90B | +110.78% | 10 |  0.00% |   0 |   0.00% |
|   +    | vote_cascading_stable_single_vote_10k |       | 189.28K | +106.94% |  0 |  0.00% |   0 |   0.00% |
|   +    | vote_cascading_stable_single_vote_1k  |       | 134.64K | +101.44% |  0 |  0.00% |   0 |   0.00% |
|   +    | vote_cascading_stable_chain_1k_5      |       | 509.36M | +101.29% |  0 |  0.00% |   0 |   0.00% |
|   +    | vote_cascading_stable_centralized_1k  |       | 200.60M | +100.61% |  1 |  0.00% |   0 |   0.00% |
|   +    | vote_cascading_stable_chain_10k_5     |       |   5.92B |  +96.70% |  5 |  0.00% |   0 |   0.00% |
|   +    | vote_cascading_stable_chain_10k_15    |       |  17.78B |  +81.05% |  5 |  0.00% |   0 |   0.00% |
|   +    | vote_cascading_stable_chain_1k_15     |       |   1.52B |  +75.52% |  0 |  0.00% |   0 |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
CSV results saved to canbench_results.csv

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@maksymar