Skip to content

Commit 80ef244

Browse files
LuciferYangdongjoon-hyun
authored andcommitted
[SPARK-52682][CORE][TESTS] Migration test case RocksDBBenchmark to microbenchmark
### What changes were proposed in this pull request? This pr migrates the test content in the test case `org.apache.spark.util.kvstore.RocksDBBenchmark` to a microbenchmark of the same name. Since the `kvstore` module is not visible to `org.apache.spark.benchmark.BenchmarkBase`, the new microbenchmark is placed in the `core` module. Additionally, this PR removes the original test case code. ### Why are the changes needed? The use of a microbenchmark is necessary for continuously monitoring the performance of `org.apache.spark.util.kvstore.RocksDB`, such as during upgrades of `rocksdbjni`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Passed Github Actions - The new microbenchmark was executed using Github Actions, and the test results were submitted. ### Was this patch authored or co-authored using generative AI tooling? No Closes #51367 from LuciferYang/Migration-RocksDBBenchmark. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 5a8cd97 commit 80ef244

File tree

4 files changed

+752
-287
lines changed

4 files changed

+752
-287
lines changed

common/kvstore/src/test/java/org/apache/spark/util/kvstore/RocksDBBenchmark.java

Lines changed: 0 additions & 287 deletions
This file was deleted.
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
================================================================================================
2+
RocksDB Lifecycle Benchmark
3+
================================================================================================
4+
5+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
6+
AMD EPYC 7763 64-Core Processor
7+
RocksDB Lifecycle Operations: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
8+
------------------------------------------------------------------------------------------------------------------------
9+
DB Creation 5 7 1 0.0 4878388.0 1.0X
10+
DB Close 0 0 0 0.0 152715.0 31.9X
11+
12+
13+
================================================================================================
14+
Sequential Operations Benchmark
15+
================================================================================================
16+
17+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
18+
AMD EPYC 7763 64-Core Processor
19+
Sequential Writes: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
20+
------------------------------------------------------------------------------------------------------------------------
21+
Indexed 23 23 1 0.0 22454.5 1.0X
22+
No Index 17 18 0 0.1 16934.6 1.3X
23+
24+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
25+
AMD EPYC 7763 64-Core Processor
26+
Sequential Updates: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
27+
------------------------------------------------------------------------------------------------------------------------
28+
Indexed 22 23 1 0.0 21573.2 1.0X
29+
No Index 10 11 1 0.1 9984.3 2.2X
30+
31+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
32+
AMD EPYC 7763 64-Core Processor
33+
Sequential Deletes: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
34+
------------------------------------------------------------------------------------------------------------------------
35+
Indexed 11 12 0 0.1 10968.0 1.0X
36+
No Index 8 8 1 0.1 7536.2 1.5X
37+
38+
39+
================================================================================================
40+
Random Operations Benchmark
41+
================================================================================================
42+
43+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
44+
AMD EPYC 7763 64-Core Processor
45+
Random Writes: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
46+
------------------------------------------------------------------------------------------------------------------------
47+
Indexed 24 26 3 0.0 23347.7 1.0X
48+
No Index 18 19 0 0.1 17813.4 1.3X
49+
50+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
51+
AMD EPYC 7763 64-Core Processor
52+
Random Updates: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
53+
------------------------------------------------------------------------------------------------------------------------
54+
Indexed 23 23 0 0.0 22277.5 1.0X
55+
No Index 11 11 0 0.1 10289.5 2.2X
56+
57+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
58+
AMD EPYC 7763 64-Core Processor
59+
Random Deletes: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
60+
------------------------------------------------------------------------------------------------------------------------
61+
Indexed 11 12 0 0.1 10970.0 1.0X
62+
No Index 8 8 0 0.1 7568.4 1.4X
63+
64+
65+
================================================================================================
66+
Natural Index Benchmark
67+
================================================================================================
68+
69+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
70+
AMD EPYC 7763 64-Core Processor
71+
Natural Index - Create Iterator: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
72+
------------------------------------------------------------------------------------------------------------------------
73+
Ascending 0 0 0 929.2 1.1 1.0X
74+
Descending 0 0 0 929.2 1.1 1.0X
75+
76+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
77+
AMD EPYC 7763 64-Core Processor
78+
Natural Index - Iteration: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
79+
------------------------------------------------------------------------------------------------------------------------
80+
Ascending 1 1 0 0.7 1358.2 1.0X
81+
Descending 1 2 0 0.7 1351.6 1.0X
82+
83+
84+
================================================================================================
85+
Ref Index Benchmark
86+
================================================================================================
87+
88+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
89+
AMD EPYC 7763 64-Core Processor
90+
Ref Index - Create Iterator: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
91+
------------------------------------------------------------------------------------------------------------------------
92+
Ascending 0 0 0 724.7 1.4 1.0X
93+
Descending 0 0 0 690.5 1.4 1.0X
94+
95+
OpenJDK 64-Bit Server VM 21.0.7+6-LTS on Linux 6.11.0-1015-azure
96+
AMD EPYC 7763 64-Core Processor
97+
Ref Index - Iteration: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
98+
------------------------------------------------------------------------------------------------------------------------
99+
Ascending 3 3 0 0.4 2675.1 1.0X
100+
Descending 3 3 1 0.4 2682.9 1.0X
101+
102+

0 commit comments

Comments
 (0)