Skip to content

Run baseline and new benchmarks; compare results, report speedups/slowdowns and significance #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dabrahams opened this issue Aug 12, 2020 · 1 comment
Assignees

Comments

@dabrahams
Copy link
Contributor

Interpreting benchmark results can be hard; in SwiftUI (and I think this was taken from something done for Swift's benchmarks) the benchmark tool would run a baseline state and new state of the code in the same process, interleaving runs, and report how much speedup or slowdown there is. Also the reports would be adorned with a mark that indicated definite speedup, definite slowdown, or statistically insignificant result. These facilities definitely helped us understand the effects we were seeing and also whether we needed to increase the iteration count or manually kill off processes, etc., to get a useful measurement. IIRC there were three ways to measure the mean result and we mostly ignored everything but the geometric mean. I can probably find out a bit more about the specifics on request. It's also very valuable to have this kind of report from CI.

@saeta
Copy link
Contributor

saeta commented Aug 19, 2020

Related: #50

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants