Benchmarking suite

[kyteware]

Is your feature request related to a problem or challenge?

Understand performance implications of changes

Describe the solution you'd like

In order for developers to productively make performance improvements, Iceberg needs a benchmarking suite.

Some things that might benefit from being benchmarked:

• Creating/reading/updating tables in different catalogs
• More granular operations, such as metadata file processing

I am happy to work on this, but I need input about:

• What tooling we want for benchmarking
• What we want benchmarked

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

[liurenjie1024]

Hi, @kyteware Thanks for you interests. Currently we have integration with datafusion, and I believe it would be a good start point. For example, you could use spark to create tpch table, and run benchmark against it.

[kyteware]

@liurenjie1024 Sounds like a good approach, do you know what approach to doing that would be legally OK? I'm getting a 404 on their EULA, and I assume you know more about them since you suggested it. Could we include their binaries to generate the TPCH table in iceberg? You need to generate a new license key every time you want to download them from their website.

[kyteware]

I've been thinking about this more and I think the TPCH idea is good, but I'm going to start off with some simple performance tests using Criterion since I'm currently working on caching for data files. The first performance tests can just be a few benchmarks on simple table reads.