RFC: Run benchmark the better way

[PsiACE]

Background

databend-benchmark uses clickhouse-driver for benchmarking. But there is a lack of a good way to present the test data. In fact, databend-benchmark is close to a rudimentary databend client.

hyperfine is a good command line benchmark tool. Supports exporting benchmark results in various formats such as json/csv, and can provide a simple markdown report.

Idea

Since Databend now supports bendctl/mysql/clickhouse-client, we can use hyperfine to take over the benchmarking directly and export the formatted data for persistence and presentation. All of this can be combined in a single workflow.

Workflow

1. Scheduled download the nightly release of databend and deploy it.
2. Use hyperfine + any client to run a specific set of tests and export the test data.
3. Persistent test data to a specific repo.
4. Based on the data, generate front-end reports and complete visualization.

TODO

• Plan the SQL suite needed for benchmarking
• Customize a workflow
• Front-end page design and data integration, ref: https://perf.rust-lang.org/

Related

• Continuous Benchmarking #3084
• Benchmark runner #3263
• Feature: Let databend-benchmark support provide reports #4322

[BohuTANG]

Great, also cc @ZhiHanZ for the Infra support :)