Reorder instruction cases in _PyEval_EvalFrameDefault

[brandtbucher]

Following our discussion yesterday, I wrote a small script for reordering the main interpreter loop in-place, as well as implementing a few simple strategies:

$ ./python.exe Tools/scripts/shuffle_ceval.py --help
usage: shuffle_ceval.py [-h] [{frequency,name,original,random,size,value}]

Reorder the cases of _PyEval_EvalFrameDefault in-place. The order is random by
default, but other orders can be specified:

./python Tools/scripts/shuffle_ceval.py frequency
    Run the PGO test suite and reorder the instructions by frequency of
    execution (requires -DDYNAMIC_EXECUTION_PROFILE).

./python Tools/scripts/shuffle_ceval.py name
    Reorder the instructions alphabetically by name.

./python Tools/scripts/shuffle_ceval.py original
    Write the instructions back in the same order (useful for debugging).

./python Tools/scripts/shuffle_ceval.py random
    Randomly shuffle the instructions (this is the default).

./python Tools/scripts/shuffle_ceval.py size
    Reorder the instructions by the size of each case.

./python Tools/scripts/shuffle_ceval.py value
    Reorder the instructions by numeric opcode value.

positional arguments:
  {frequency,name,original,random,size,value}

options:
  -h, --help            show this help message and exit

You can find it on the shuffle-ceval branch of my fork. I wanted to stop here and gather feedback before figuring out the next steps with this project.

[gvanrossum]

A suggestion: put the shuffling script in the faster-cpython/tools repo. That way you don't have to work on a branch or get code review to improve your script. I'll give you access.

[brandtbucher]

Thanks!

I did have to make some tiny changes to the main loop to get this to work, though. Should those be upstreamed?

[gvanrossum]

Can you send a separate PR for those? I'm a little bit worried that the change to JUMP_IF_NOT_EXC_MATCH might affect the generated code and possibly its performance? (But @markshannon should review that.)

[gvanrossum]

I also realize that my suggestion actually makes your workflow more complicated, so feel free to ignore it.

[markshannon]

The changes to ceval.c look fine, if you want to upstream them.

[brandtbucher]

Cool. Then I'll try benchmarking a random shuffle and see if that creates any significant differences.

[kmod]

The advice I've heard is to order code by density (execution count divided by instruction bytes), though I was under the impression that PGO did branch reordering and would do this sort of thing automatically.

[brandtbucher]

The advice I've heard is to order code by density (execution count divided by instruction bytes)...

Interesting... that's easy enough to add to the script.

...though I was under the impression that PGO did branch reordering and would do this sort of thing automatically.

Yeah, I'm expecting that the difference between two orders will be negligible. We'll see.

[brandtbucher]

The results are in, and... nothing significant seems to happen when reordering the ceval cases (as expected).

I benchmarked the following changes against main:

• main-again: no changes, just running the same exact commit again to get an idea of our noise floor.
• shuffle-ceval-random-a: a random shuffle of _PyEval_EvalFrameDefault.
• shuffle-ceval-random-b: ditto.
• shuffle-ceval-random-c: ditto.
• shuffle-ceval-random-d: ditto.
• shuffle-ceval-density: reorder instructions by hit count during PGO, divided by approximate source size.

Geometric mean
==============

main-again: 1.00x slower
shuffle-ceval-random-a: 1.01x slower
shuffle-ceval-random-b: 1.01x slower
shuffle-ceval-random-c: 1.00x faster
shuffle-ceval-random-d: 1.00x slower
shuffle-ceval-density: 1.00x slower

2to3
====

Mean +- std dev: [main] 269 ms +- 1 ms -> [main-again] 270 ms +- 1 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 269 ms +- 1 ms -> [shuffle-ceval-random-a] 268 ms +- 1 ms: 1.00x faster
Mean +- std dev: [main] 269 ms +- 1 ms -> [shuffle-ceval-random-b] 269 ms +- 1 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 269 ms +- 1 ms -> [shuffle-ceval-random-c] 268 ms +- 1 ms: 1.01x faster
Mean +- std dev: [main] 269 ms +- 1 ms -> [shuffle-ceval-random-d] 269 ms +- 1 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 269 ms +- 1 ms -> [shuffle-ceval-density] 268 ms +- 1 ms: 1.01x faster

chameleon
=========

Mean +- std dev: [main] 7.66 ms +- 0.07 ms -> [main-again] 7.67 ms +- 0.10 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 7.66 ms +- 0.07 ms -> [shuffle-ceval-random-a] 7.77 ms +- 0.05 ms: 1.01x slower
Mean +- std dev: [main] 7.66 ms +- 0.07 ms -> [shuffle-ceval-random-b] 7.88 ms +- 0.13 ms: 1.03x slower
Mean +- std dev: [main] 7.66 ms +- 0.07 ms -> [shuffle-ceval-random-c] 7.68 ms +- 0.11 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 7.66 ms +- 0.07 ms -> [shuffle-ceval-random-d] 7.69 ms +- 0.06 ms: 1.00x slower
Mean +- std dev: [main] 7.66 ms +- 0.07 ms -> [shuffle-ceval-density] 7.84 ms +- 0.09 ms: 1.02x slower

chaos
=====

Mean +- std dev: [main] 79.0 ms +- 1.0 ms -> [main-again] 78.3 ms +- 0.8 ms: 1.01x faster
Mean +- std dev: [main] 79.0 ms +- 1.0 ms -> [shuffle-ceval-random-a] 79.3 ms +- 1.0 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 79.0 ms +- 1.0 ms -> [shuffle-ceval-random-b] 78.8 ms +- 0.8 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 79.0 ms +- 1.0 ms -> [shuffle-ceval-random-c] 78.1 ms +- 0.9 ms: 1.01x faster
Mean +- std dev: [main] 79.0 ms +- 1.0 ms -> [shuffle-ceval-random-d] 78.1 ms +- 0.8 ms: 1.01x faster
Mean +- std dev: [main] 79.0 ms +- 1.0 ms -> [shuffle-ceval-density] 78.7 ms +- 0.9 ms: 1.00x faster

crypto_pyaes
============

Mean +- std dev: [main] 88.4 ms +- 0.5 ms -> [main-again] 87.9 ms +- 1.0 ms: 1.01x faster
Mean +- std dev: [main] 88.4 ms +- 0.5 ms -> [shuffle-ceval-random-a] 88.6 ms +- 0.9 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 88.4 ms +- 0.5 ms -> [shuffle-ceval-random-b] 88.3 ms +- 1.1 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 88.4 ms +- 0.5 ms -> [shuffle-ceval-random-c] 87.2 ms +- 0.7 ms: 1.01x faster
Mean +- std dev: [main] 88.4 ms +- 0.5 ms -> [shuffle-ceval-random-d] 86.2 ms +- 0.7 ms: 1.03x faster
Mean +- std dev: [main] 88.4 ms +- 0.5 ms -> [shuffle-ceval-density] 87.5 ms +- 0.8 ms: 1.01x faster

deltablue
=========

Mean +- std dev: [main] 4.90 ms +- 0.05 ms -> [main-again] 4.92 ms +- 0.06 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 4.90 ms +- 0.05 ms -> [shuffle-ceval-random-a] 4.95 ms +- 0.07 ms: 1.01x slower
Mean +- std dev: [main] 4.90 ms +- 0.05 ms -> [shuffle-ceval-random-b] 4.91 ms +- 0.07 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 4.90 ms +- 0.05 ms -> [shuffle-ceval-random-c] 4.88 ms +- 0.07 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 4.90 ms +- 0.05 ms -> [shuffle-ceval-random-d] 4.94 ms +- 0.06 ms: 1.01x slower
Mean +- std dev: [main] 4.90 ms +- 0.05 ms -> [shuffle-ceval-density] 4.96 ms +- 0.07 ms: 1.01x slower

django_template
===============

Mean +- std dev: [main] 39.0 ms +- 0.4 ms -> [main-again] 39.2 ms +- 0.3 ms: 1.01x slower
Mean +- std dev: [main] 39.0 ms +- 0.4 ms -> [shuffle-ceval-random-a] 38.9 ms +- 0.4 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 39.0 ms +- 0.4 ms -> [shuffle-ceval-random-b] 38.7 ms +- 0.3 ms: 1.01x faster
Mean +- std dev: [main] 39.0 ms +- 0.4 ms -> [shuffle-ceval-random-c] 38.7 ms +- 0.4 ms: 1.01x faster
Mean +- std dev: [main] 39.0 ms +- 0.4 ms -> [shuffle-ceval-random-d] 39.2 ms +- 0.4 ms: 1.01x slower
Mean +- std dev: [main] 39.0 ms +- 0.4 ms -> [shuffle-ceval-density] 38.6 ms +- 0.5 ms: 1.01x faster

dulwich_log
===========

Mean +- std dev: [main] 68.6 ms +- 0.4 ms -> [main-again] 68.8 ms +- 0.5 ms: 1.00x slower
Mean +- std dev: [main] 68.6 ms +- 0.4 ms -> [shuffle-ceval-random-a] 68.1 ms +- 0.4 ms: 1.01x faster
Mean +- std dev: [main] 68.6 ms +- 0.4 ms -> [shuffle-ceval-random-b] 68.3 ms +- 0.4 ms: 1.00x faster
Mean +- std dev: [main] 68.6 ms +- 0.4 ms -> [shuffle-ceval-random-c] 68.4 ms +- 0.7 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 68.6 ms +- 0.4 ms -> [shuffle-ceval-random-d] 68.8 ms +- 0.3 ms: 1.00x slower
Mean +- std dev: [main] 68.6 ms +- 0.4 ms -> [shuffle-ceval-density] 68.2 ms +- 0.4 ms: 1.01x faster

fannkuch
========

Mean +- std dev: [main] 421 ms +- 2 ms -> [main-again] 426 ms +- 5 ms: 1.01x slower
Mean +- std dev: [main] 421 ms +- 2 ms -> [shuffle-ceval-random-a] 435 ms +- 4 ms: 1.03x slower
Mean +- std dev: [main] 421 ms +- 2 ms -> [shuffle-ceval-random-b] 434 ms +- 7 ms: 1.03x slower
Mean +- std dev: [main] 421 ms +- 2 ms -> [shuffle-ceval-random-c] 420 ms +- 4 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 421 ms +- 2 ms -> [shuffle-ceval-random-d] 423 ms +- 4 ms: 1.00x slower
Mean +- std dev: [main] 421 ms +- 2 ms -> [shuffle-ceval-density] 428 ms +- 3 ms: 1.02x slower

float
=====

Mean +- std dev: [main] 84.7 ms +- 1.0 ms -> [main-again] 85.0 ms +- 0.8 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 84.7 ms +- 1.0 ms -> [shuffle-ceval-random-a] 84.2 ms +- 0.9 ms: 1.01x faster
Mean +- std dev: [main] 84.7 ms +- 1.0 ms -> [shuffle-ceval-random-b] 84.6 ms +- 0.8 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 84.7 ms +- 1.0 ms -> [shuffle-ceval-random-c] 84.5 ms +- 0.9 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 84.7 ms +- 1.0 ms -> [shuffle-ceval-random-d] 84.4 ms +- 0.9 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 84.7 ms +- 1.0 ms -> [shuffle-ceval-density] 85.6 ms +- 0.8 ms: 1.01x slower

go
==

Mean +- std dev: [main] 171 ms +- 2 ms -> [main-again] 172 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 171 ms +- 2 ms -> [shuffle-ceval-random-a] 169 ms +- 2 ms: 1.01x faster
Mean +- std dev: [main] 171 ms +- 2 ms -> [shuffle-ceval-random-b] 170 ms +- 1 ms: 1.01x faster
Mean +- std dev: [main] 171 ms +- 2 ms -> [shuffle-ceval-random-c] 169 ms +- 2 ms: 1.01x faster
Mean +- std dev: [main] 171 ms +- 2 ms -> [shuffle-ceval-random-d] 171 ms +- 3 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 171 ms +- 2 ms -> [shuffle-ceval-density] 171 ms +- 1 ms: 1.00x slower
Not significant!

hexiom
======

Mean +- std dev: [main] 7.64 ms +- 0.05 ms -> [main-again] 7.64 ms +- 0.05 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 7.64 ms +- 0.05 ms -> [shuffle-ceval-random-a] 7.81 ms +- 0.06 ms: 1.02x slower
Mean +- std dev: [main] 7.64 ms +- 0.05 ms -> [shuffle-ceval-random-b] 7.79 ms +- 0.04 ms: 1.02x slower
Mean +- std dev: [main] 7.64 ms +- 0.05 ms -> [shuffle-ceval-random-c] 7.63 ms +- 0.07 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 7.64 ms +- 0.05 ms -> [shuffle-ceval-random-d] 7.68 ms +- 0.07 ms: 1.01x slower
Mean +- std dev: [main] 7.64 ms +- 0.05 ms -> [shuffle-ceval-density] 7.61 ms +- 0.05 ms: 1.00x faster

json_dumps
==========

Mean +- std dev: [main] 12.5 ms +- 0.2 ms -> [main-again] 12.6 ms +- 0.1 ms: 1.01x slower
Mean +- std dev: [main] 12.5 ms +- 0.2 ms -> [shuffle-ceval-random-a] 12.8 ms +- 0.2 ms: 1.03x slower
Mean +- std dev: [main] 12.5 ms +- 0.2 ms -> [shuffle-ceval-random-b] 12.6 ms +- 0.1 ms: 1.01x slower
Mean +- std dev: [main] 12.5 ms +- 0.2 ms -> [shuffle-ceval-random-c] 12.4 ms +- 0.1 ms: 1.01x faster
Mean +- std dev: [main] 12.5 ms +- 0.2 ms -> [shuffle-ceval-random-d] 12.5 ms +- 0.1 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 12.5 ms +- 0.2 ms -> [shuffle-ceval-density] 12.7 ms +- 0.1 ms: 1.02x slower

json_loads
==========

Mean +- std dev: [main] 25.3 us +- 0.7 us -> [main-again] 25.9 us +- 0.3 us: 1.02x slower
Mean +- std dev: [main] 25.3 us +- 0.7 us -> [shuffle-ceval-random-a] 25.8 us +- 0.4 us: 1.02x slower
Mean +- std dev: [main] 25.3 us +- 0.7 us -> [shuffle-ceval-random-b] 25.9 us +- 0.3 us: 1.02x slower
Mean +- std dev: [main] 25.3 us +- 0.7 us -> [shuffle-ceval-random-c] 25.6 us +- 0.2 us: 1.01x slower
Mean +- std dev: [main] 25.3 us +- 0.7 us -> [shuffle-ceval-random-d] 25.8 us +- 0.5 us: 1.02x slower
Mean +- std dev: [main] 25.3 us +- 0.7 us -> [shuffle-ceval-density] 25.9 us +- 0.9 us: 1.02x slower

logging_format
==============

Mean +- std dev: [main] 6.82 us +- 0.08 us -> [main-again] 7.00 us +- 0.12 us: 1.03x slower
Mean +- std dev: [main] 6.82 us +- 0.08 us -> [shuffle-ceval-random-a] 6.73 us +- 0.10 us: 1.01x faster
Mean +- std dev: [main] 6.82 us +- 0.08 us -> [shuffle-ceval-random-b] 6.87 us +- 0.07 us: 1.01x slower
Mean +- std dev: [main] 6.82 us +- 0.08 us -> [shuffle-ceval-random-c] 6.79 us +- 0.09 us: 1.00x faster
Not significant!
Mean +- std dev: [main] 6.82 us +- 0.08 us -> [shuffle-ceval-random-d] 6.86 us +- 0.07 us: 1.01x slower
Mean +- std dev: [main] 6.82 us +- 0.08 us -> [shuffle-ceval-density] 6.77 us +- 0.10 us: 1.01x faster

logging_silent
==============

Mean +- std dev: [main] 115 ns +- 2 ns -> [main-again] 116 ns +- 1 ns: 1.01x slower
Mean +- std dev: [main] 115 ns +- 2 ns -> [shuffle-ceval-random-a] 113 ns +- 1 ns: 1.02x faster
Mean +- std dev: [main] 115 ns +- 2 ns -> [shuffle-ceval-random-b] 114 ns +- 2 ns: 1.01x faster
Mean +- std dev: [main] 115 ns +- 2 ns -> [shuffle-ceval-random-c] 115 ns +- 1 ns: 1.00x faster
Not significant!
Mean +- std dev: [main] 115 ns +- 2 ns -> [shuffle-ceval-random-d] 117 ns +- 2 ns: 1.01x slower
Mean +- std dev: [main] 115 ns +- 2 ns -> [shuffle-ceval-density] 113 ns +- 2 ns: 1.02x faster

logging_simple
==============

Mean +- std dev: [main] 6.10 us +- 0.08 us -> [main-again] 6.16 us +- 0.09 us: 1.01x slower
Mean +- std dev: [main] 6.10 us +- 0.08 us -> [shuffle-ceval-random-a] 6.01 us +- 0.08 us: 1.01x faster
Mean +- std dev: [main] 6.10 us +- 0.08 us -> [shuffle-ceval-random-b] 6.19 us +- 0.07 us: 1.01x slower
Mean +- std dev: [main] 6.10 us +- 0.08 us -> [shuffle-ceval-random-c] 6.08 us +- 0.08 us: 1.00x faster
Not significant!
Mean +- std dev: [main] 6.10 us +- 0.08 us -> [shuffle-ceval-random-d] 6.18 us +- 0.09 us: 1.01x slower
Mean +- std dev: [main] 6.10 us +- 0.08 us -> [shuffle-ceval-density] 6.11 us +- 0.12 us: 1.00x slower
Not significant!

mako
====

Mean +- std dev: [main] 12.7 ms +- 0.1 ms -> [main-again] 12.8 ms +- 0.1 ms: 1.01x slower
Mean +- std dev: [main] 12.7 ms +- 0.1 ms -> [shuffle-ceval-random-a] 13.1 ms +- 0.1 ms: 1.03x slower
Mean +- std dev: [main] 12.7 ms +- 0.1 ms -> [shuffle-ceval-random-b] 13.2 ms +- 0.3 ms: 1.04x slower
Mean +- std dev: [main] 12.7 ms +- 0.1 ms -> [shuffle-ceval-random-c] 12.8 ms +- 0.1 ms: 1.01x slower
Mean +- std dev: [main] 12.7 ms +- 0.1 ms -> [shuffle-ceval-random-d] 12.8 ms +- 0.1 ms: 1.01x slower
Mean +- std dev: [main] 12.7 ms +- 0.1 ms -> [shuffle-ceval-density] 12.9 ms +- 0.0 ms: 1.01x slower

meteor_contest
==============

Mean +- std dev: [main] 107 ms +- 1 ms -> [main-again] 109 ms +- 4 ms: 1.02x slower
Mean +- std dev: [main] 107 ms +- 1 ms -> [shuffle-ceval-random-a] 108 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 107 ms +- 1 ms -> [shuffle-ceval-random-b] 108 ms +- 3 ms: 1.01x slower
Mean +- std dev: [main] 107 ms +- 1 ms -> [shuffle-ceval-random-c] 107 ms +- 2 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 107 ms +- 1 ms -> [shuffle-ceval-random-d] 107 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 107 ms +- 1 ms -> [shuffle-ceval-density] 107 ms +- 2 ms: 1.00x slower
Not significant!

nbody
=====

Mean +- std dev: [main] 113 ms +- 1 ms -> [main-again] 114 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 113 ms +- 1 ms -> [shuffle-ceval-random-a] 115 ms +- 1 ms: 1.02x slower
Mean +- std dev: [main] 113 ms +- 1 ms -> [shuffle-ceval-random-b] 114 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 113 ms +- 1 ms -> [shuffle-ceval-random-c] 115 ms +- 2 ms: 1.02x slower
Mean +- std dev: [main] 113 ms +- 1 ms -> [shuffle-ceval-random-d] 115 ms +- 2 ms: 1.02x slower
Mean +- std dev: [main] 113 ms +- 1 ms -> [shuffle-ceval-density] 114 ms +- 1 ms: 1.01x slower

nqueens
=======

Mean +- std dev: [main] 89.6 ms +- 1.3 ms -> [main-again] 88.3 ms +- 0.6 ms: 1.01x faster
Mean +- std dev: [main] 89.6 ms +- 1.3 ms -> [shuffle-ceval-random-a] 89.7 ms +- 0.8 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 89.6 ms +- 1.3 ms -> [shuffle-ceval-random-b] 87.9 ms +- 0.9 ms: 1.02x faster
Mean +- std dev: [main] 89.6 ms +- 1.3 ms -> [shuffle-ceval-random-c] 87.8 ms +- 1.5 ms: 1.02x faster
Mean +- std dev: [main] 89.6 ms +- 1.3 ms -> [shuffle-ceval-random-d] 93.3 ms +- 1.1 ms: 1.04x slower
Mean +- std dev: [main] 89.6 ms +- 1.3 ms -> [shuffle-ceval-density] 89.5 ms +- 0.9 ms: 1.00x faster
Not significant!

pathlib
=======

Mean +- std dev: [main] 19.2 ms +- 0.3 ms -> [main-again] 19.2 ms +- 0.2 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 19.2 ms +- 0.3 ms -> [shuffle-ceval-random-a] 19.2 ms +- 0.2 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 19.2 ms +- 0.3 ms -> [shuffle-ceval-random-b] 18.9 ms +- 0.2 ms: 1.02x faster
Mean +- std dev: [main] 19.2 ms +- 0.3 ms -> [shuffle-ceval-random-c] 18.9 ms +- 0.2 ms: 1.02x faster
Mean +- std dev: [main] 19.2 ms +- 0.3 ms -> [shuffle-ceval-random-d] 19.3 ms +- 0.2 ms: 1.01x slower
Mean +- std dev: [main] 19.2 ms +- 0.3 ms -> [shuffle-ceval-density] 19.2 ms +- 0.1 ms: 1.00x faster
Not significant!

pickle
======

Mean +- std dev: [main] 9.87 us +- 0.09 us -> [main-again] 10.0 us +- 0.1 us: 1.01x slower
Mean +- std dev: [main] 9.87 us +- 0.09 us -> [shuffle-ceval-random-a] 10.4 us +- 0.1 us: 1.05x slower
Mean +- std dev: [main] 9.87 us +- 0.09 us -> [shuffle-ceval-random-b] 10.4 us +- 0.1 us: 1.05x slower
Mean +- std dev: [main] 9.87 us +- 0.09 us -> [shuffle-ceval-random-c] 9.86 us +- 0.10 us: 1.00x faster
Not significant!
Mean +- std dev: [main] 9.87 us +- 0.09 us -> [shuffle-ceval-random-d] 10.1 us +- 0.1 us: 1.03x slower
Mean +- std dev: [main] 9.87 us +- 0.09 us -> [shuffle-ceval-density] 10.3 us +- 0.1 us: 1.04x slower

pickle_dict
===========

Mean +- std dev: [main] 27.2 us +- 0.2 us -> [main-again] 27.3 us +- 0.2 us: 1.01x slower
Mean +- std dev: [main] 27.2 us +- 0.2 us -> [shuffle-ceval-random-a] 28.4 us +- 0.1 us: 1.05x slower
Mean +- std dev: [main] 27.2 us +- 0.2 us -> [shuffle-ceval-random-b] 28.8 us +- 0.1 us: 1.06x slower
Mean +- std dev: [main] 27.2 us +- 0.2 us -> [shuffle-ceval-random-c] 27.7 us +- 0.1 us: 1.02x slower
Mean +- std dev: [main] 27.2 us +- 0.2 us -> [shuffle-ceval-random-d] 27.4 us +- 0.1 us: 1.01x slower
Mean +- std dev: [main] 27.2 us +- 0.2 us -> [shuffle-ceval-density] 28.5 us +- 0.1 us: 1.05x slower

pickle_list
===========

Mean +- std dev: [main] 4.41 us +- 0.06 us -> [main-again] 4.53 us +- 0.04 us: 1.03x slower
Mean +- std dev: [main] 4.41 us +- 0.06 us -> [shuffle-ceval-random-a] 4.52 us +- 0.06 us: 1.02x slower
Mean +- std dev: [main] 4.41 us +- 0.06 us -> [shuffle-ceval-random-b] 4.57 us +- 0.05 us: 1.04x slower
Mean +- std dev: [main] 4.41 us +- 0.06 us -> [shuffle-ceval-random-c] 4.34 us +- 0.05 us: 1.02x faster
Mean +- std dev: [main] 4.41 us +- 0.06 us -> [shuffle-ceval-random-d] 4.48 us +- 0.05 us: 1.01x slower
Mean +- std dev: [main] 4.41 us +- 0.06 us -> [shuffle-ceval-density] 4.57 us +- 0.04 us: 1.03x slower

pickle_pure_python
==================

Mean +- std dev: [main] 379 us +- 3 us -> [main-again] 376 us +- 3 us: 1.01x faster
Mean +- std dev: [main] 379 us +- 3 us -> [shuffle-ceval-random-a] 380 us +- 6 us: 1.00x slower
Not significant!
Mean +- std dev: [main] 379 us +- 3 us -> [shuffle-ceval-random-b] 379 us +- 4 us: 1.00x slower
Not significant!
Mean +- std dev: [main] 379 us +- 3 us -> [shuffle-ceval-random-c] 379 us +- 4 us: 1.00x slower
Not significant!
Mean +- std dev: [main] 379 us +- 3 us -> [shuffle-ceval-random-d] 378 us +- 3 us: 1.00x faster
Not significant!
Mean +- std dev: [main] 379 us +- 3 us -> [shuffle-ceval-density] 379 us +- 5 us: 1.00x slower
Not significant!

pidigits
========

Mean +- std dev: [main] 188 ms +- 0 ms -> [main-again] 208 ms +- 0 ms: 1.11x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-a] 189 ms +- 0 ms: 1.00x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-b] 209 ms +- 0 ms: 1.11x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-c] 188 ms +- 0 ms: 1.00x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-d] 188 ms +- 0 ms: 1.00x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-density] 205 ms +- 0 ms: 1.09x slower

pyflate
=======

Mean +- std dev: [main] 531 ms +- 4 ms -> [main-again] 536 ms +- 6 ms: 1.01x slower
Mean +- std dev: [main] 531 ms +- 4 ms -> [shuffle-ceval-random-a] 544 ms +- 6 ms: 1.02x slower
Mean +- std dev: [main] 531 ms +- 4 ms -> [shuffle-ceval-random-b] 538 ms +- 4 ms: 1.01x slower
Mean +- std dev: [main] 531 ms +- 4 ms -> [shuffle-ceval-random-c] 528 ms +- 7 ms: 1.01x faster
Mean +- std dev: [main] 531 ms +- 4 ms -> [shuffle-ceval-random-d] 542 ms +- 5 ms: 1.02x slower
Mean +- std dev: [main] 531 ms +- 4 ms -> [shuffle-ceval-density] 533 ms +- 7 ms: 1.00x slower

python_startup
==============

Mean +- std dev: [main] 8.46 ms +- 0.01 ms -> [main-again] 8.45 ms +- 0.01 ms: 1.00x faster
Mean +- std dev: [main] 8.46 ms +- 0.01 ms -> [shuffle-ceval-random-a] 8.41 ms +- 0.01 ms: 1.01x faster
Mean +- std dev: [main] 8.46 ms +- 0.01 ms -> [shuffle-ceval-random-b] 8.45 ms +- 0.01 ms: 1.00x faster
Mean +- std dev: [main] 8.46 ms +- 0.01 ms -> [shuffle-ceval-random-c] 8.47 ms +- 0.01 ms: 1.00x slower
Mean +- std dev: [main] 8.46 ms +- 0.01 ms -> [shuffle-ceval-random-d] 8.45 ms +- 0.01 ms: 1.00x faster
Mean +- std dev: [main] 8.46 ms +- 0.01 ms -> [shuffle-ceval-density] 8.48 ms +- 0.01 ms: 1.00x slower

python_startup_no_site
======================

Mean +- std dev: [main] 5.85 ms +- 0.01 ms -> [main-again] 5.84 ms +- 0.00 ms: 1.00x faster
Mean +- std dev: [main] 5.85 ms +- 0.01 ms -> [shuffle-ceval-random-a] 5.82 ms +- 0.00 ms: 1.01x faster
Mean +- std dev: [main] 5.85 ms +- 0.01 ms -> [shuffle-ceval-random-b] 5.84 ms +- 0.00 ms: 1.00x faster
Mean +- std dev: [main] 5.85 ms +- 0.01 ms -> [shuffle-ceval-random-c] 5.85 ms +- 0.00 ms: 1.00x slower
Mean +- std dev: [main] 5.85 ms +- 0.01 ms -> [shuffle-ceval-random-d] 5.84 ms +- 0.01 ms: 1.00x faster
Mean +- std dev: [main] 5.85 ms +- 0.01 ms -> [shuffle-ceval-density] 5.86 ms +- 0.00 ms: 1.00x slower

raytrace
========

Mean +- std dev: [main] 326 ms +- 2 ms -> [main-again] 331 ms +- 3 ms: 1.02x slower
Mean +- std dev: [main] 326 ms +- 2 ms -> [shuffle-ceval-random-a] 326 ms +- 2 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 326 ms +- 2 ms -> [shuffle-ceval-random-b] 328 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 326 ms +- 2 ms -> [shuffle-ceval-random-c] 330 ms +- 3 ms: 1.01x slower
Mean +- std dev: [main] 326 ms +- 2 ms -> [shuffle-ceval-random-d] 327 ms +- 2 ms: 1.00x slower
Mean +- std dev: [main] 326 ms +- 2 ms -> [shuffle-ceval-density] 327 ms +- 3 ms: 1.00x slower

regex_compile
=============

Mean +- std dev: [main] 151 ms +- 1 ms -> [main-again] 151 ms +- 1 ms: 1.01x slower
Mean +- std dev: [main] 151 ms +- 1 ms -> [shuffle-ceval-random-a] 151 ms +- 1 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 151 ms +- 1 ms -> [shuffle-ceval-random-b] 152 ms +- 1 ms: 1.01x slower
Mean +- std dev: [main] 151 ms +- 1 ms -> [shuffle-ceval-random-c] 149 ms +- 1 ms: 1.01x faster
Mean +- std dev: [main] 151 ms +- 1 ms -> [shuffle-ceval-random-d] 150 ms +- 1 ms: 1.01x faster
Mean +- std dev: [main] 151 ms +- 1 ms -> [shuffle-ceval-density] 150 ms +- 1 ms: 1.01x faster

regex_dna
=========

Mean +- std dev: [main] 214 ms +- 1 ms -> [main-again] 212 ms +- 2 ms: 1.01x faster
Mean +- std dev: [main] 214 ms +- 1 ms -> [shuffle-ceval-random-a] 209 ms +- 3 ms: 1.03x faster
Mean +- std dev: [main] 214 ms +- 1 ms -> [shuffle-ceval-random-b] 210 ms +- 2 ms: 1.02x faster
Mean +- std dev: [main] 214 ms +- 1 ms -> [shuffle-ceval-random-c] 210 ms +- 5 ms: 1.02x faster
Mean +- std dev: [main] 214 ms +- 1 ms -> [shuffle-ceval-random-d] 216 ms +- 1 ms: 1.01x slower
Mean +- std dev: [main] 214 ms +- 1 ms -> [shuffle-ceval-density] 206 ms +- 3 ms: 1.04x faster

regex_effbot
============

Mean +- std dev: [main] 3.31 ms +- 0.05 ms -> [main-again] 3.11 ms +- 0.08 ms: 1.06x faster
Mean +- std dev: [main] 3.31 ms +- 0.05 ms -> [shuffle-ceval-random-a] 3.19 ms +- 0.06 ms: 1.04x faster
Mean +- std dev: [main] 3.31 ms +- 0.05 ms -> [shuffle-ceval-random-b] 3.13 ms +- 0.05 ms: 1.06x faster
Mean +- std dev: [main] 3.31 ms +- 0.05 ms -> [shuffle-ceval-random-c] 3.18 ms +- 0.07 ms: 1.04x faster
Mean +- std dev: [main] 3.31 ms +- 0.05 ms -> [shuffle-ceval-random-d] 3.28 ms +- 0.03 ms: 1.01x faster
Mean +- std dev: [main] 3.31 ms +- 0.05 ms -> [shuffle-ceval-density] 3.18 ms +- 0.11 ms: 1.04x faster

regex_v8
========

Mean +- std dev: [main] 24.3 ms +- 0.1 ms -> [main-again] 24.1 ms +- 0.5 ms: 1.01x faster
Mean +- std dev: [main] 24.3 ms +- 0.1 ms -> [shuffle-ceval-random-a] 23.8 ms +- 0.1 ms: 1.02x faster
Mean +- std dev: [main] 24.3 ms +- 0.1 ms -> [shuffle-ceval-random-b] 23.7 ms +- 0.2 ms: 1.02x faster
Mean +- std dev: [main] 24.3 ms +- 0.1 ms -> [shuffle-ceval-random-c] 23.6 ms +- 0.1 ms: 1.03x faster
Mean +- std dev: [main] 24.3 ms +- 0.1 ms -> [shuffle-ceval-random-d] 25.1 ms +- 0.3 ms: 1.03x slower
Mean +- std dev: [main] 24.3 ms +- 0.1 ms -> [shuffle-ceval-density] 23.8 ms +- 0.2 ms: 1.02x faster

richards
========

Mean +- std dev: [main] 55.3 ms +- 0.8 ms -> [main-again] 55.5 ms +- 1.6 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 55.3 ms +- 0.8 ms -> [shuffle-ceval-random-a] 56.1 ms +- 1.8 ms: 1.01x slower
Mean +- std dev: [main] 55.3 ms +- 0.8 ms -> [shuffle-ceval-random-b] 56.6 ms +- 0.7 ms: 1.02x slower
Mean +- std dev: [main] 55.3 ms +- 0.8 ms -> [shuffle-ceval-random-c] 55.6 ms +- 0.6 ms: 1.01x slower
Mean +- std dev: [main] 55.3 ms +- 0.8 ms -> [shuffle-ceval-random-d] 55.4 ms +- 0.7 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 55.3 ms +- 0.8 ms -> [shuffle-ceval-density] 57.0 ms +- 1.1 ms: 1.03x slower

scimark_fft
===========

Mean +- std dev: [main] 350 ms +- 3 ms -> [main-again] 345 ms +- 3 ms: 1.01x faster
Mean +- std dev: [main] 350 ms +- 3 ms -> [shuffle-ceval-random-a] 356 ms +- 3 ms: 1.02x slower
Mean +- std dev: [main] 350 ms +- 3 ms -> [shuffle-ceval-random-b] 361 ms +- 3 ms: 1.03x slower
Mean +- std dev: [main] 350 ms +- 3 ms -> [shuffle-ceval-random-c] 341 ms +- 3 ms: 1.03x faster
Mean +- std dev: [main] 350 ms +- 3 ms -> [shuffle-ceval-random-d] 351 ms +- 2 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 350 ms +- 3 ms -> [shuffle-ceval-density] 350 ms +- 4 ms: 1.00x faster
Not significant!

scimark_lu
==========

Mean +- std dev: [main] 150 ms +- 3 ms -> [main-again] 151 ms +- 6 ms: 1.01x slower
Not significant!
Mean +- std dev: [main] 150 ms +- 3 ms -> [shuffle-ceval-random-a] 151 ms +- 5 ms: 1.01x slower
Not significant!
Mean +- std dev: [main] 150 ms +- 3 ms -> [shuffle-ceval-random-b] 152 ms +- 4 ms: 1.01x slower
Mean +- std dev: [main] 150 ms +- 3 ms -> [shuffle-ceval-random-c] 152 ms +- 5 ms: 1.02x slower
Mean +- std dev: [main] 150 ms +- 3 ms -> [shuffle-ceval-random-d] 148 ms +- 5 ms: 1.01x faster
Mean +- std dev: [main] 150 ms +- 3 ms -> [shuffle-ceval-density] 152 ms +- 4 ms: 1.02x slower

scimark_monte_carlo
===================

Mean +- std dev: [main] 78.7 ms +- 0.9 ms -> [main-again] 80.1 ms +- 1.8 ms: 1.02x slower
Mean +- std dev: [main] 78.7 ms +- 0.9 ms -> [shuffle-ceval-random-a] 81.9 ms +- 2.8 ms: 1.04x slower
Mean +- std dev: [main] 78.7 ms +- 0.9 ms -> [shuffle-ceval-random-b] 78.4 ms +- 1.3 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 78.7 ms +- 0.9 ms -> [shuffle-ceval-random-c] 78.2 ms +- 1.6 ms: 1.01x faster
Mean +- std dev: [main] 78.7 ms +- 0.9 ms -> [shuffle-ceval-random-d] 77.1 ms +- 0.8 ms: 1.02x faster
Mean +- std dev: [main] 78.7 ms +- 0.9 ms -> [shuffle-ceval-density] 78.7 ms +- 1.9 ms: 1.00x faster
Not significant!

scimark_sor
===========

Mean +- std dev: [main] 156 ms +- 2 ms -> [main-again] 158 ms +- 4 ms: 1.01x slower
Mean +- std dev: [main] 156 ms +- 2 ms -> [shuffle-ceval-random-a] 160 ms +- 5 ms: 1.03x slower
Mean +- std dev: [main] 156 ms +- 2 ms -> [shuffle-ceval-random-b] 158 ms +- 4 ms: 1.01x slower
Mean +- std dev: [main] 156 ms +- 2 ms -> [shuffle-ceval-random-c] 157 ms +- 3 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 156 ms +- 2 ms -> [shuffle-ceval-random-d] 157 ms +- 1 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 156 ms +- 2 ms -> [shuffle-ceval-density] 155 ms +- 3 ms: 1.01x faster
Not significant!

scimark_sparse_mat_mult
=======================

Mean +- std dev: [main] 4.90 ms +- 0.07 ms -> [main-again] 4.94 ms +- 0.02 ms: 1.01x slower
Mean +- std dev: [main] 4.90 ms +- 0.07 ms -> [shuffle-ceval-random-a] 4.83 ms +- 0.10 ms: 1.01x faster
Mean +- std dev: [main] 4.90 ms +- 0.07 ms -> [shuffle-ceval-random-b] 4.86 ms +- 0.03 ms: 1.01x faster
Mean +- std dev: [main] 4.90 ms +- 0.07 ms -> [shuffle-ceval-random-c] 4.90 ms +- 0.11 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 4.90 ms +- 0.07 ms -> [shuffle-ceval-random-d] 4.80 ms +- 0.11 ms: 1.02x faster
Mean +- std dev: [main] 4.90 ms +- 0.07 ms -> [shuffle-ceval-density] 4.68 ms +- 0.03 ms: 1.05x faster

spectral_norm
=============

Mean +- std dev: [main] 114 ms +- 3 ms -> [main-again] 111 ms +- 2 ms: 1.03x faster
Mean +- std dev: [main] 114 ms +- 3 ms -> [shuffle-ceval-random-a] 118 ms +- 2 ms: 1.03x slower
Mean +- std dev: [main] 114 ms +- 3 ms -> [shuffle-ceval-random-b] 120 ms +- 1 ms: 1.05x slower
Mean +- std dev: [main] 114 ms +- 3 ms -> [shuffle-ceval-random-c] 111 ms +- 2 ms: 1.03x faster
Mean +- std dev: [main] 114 ms +- 3 ms -> [shuffle-ceval-random-d] 115 ms +- 0 ms: 1.01x slower
Mean +- std dev: [main] 114 ms +- 3 ms -> [shuffle-ceval-density] 110 ms +- 2 ms: 1.04x faster

sqlalchemy_declarative
======================

Mean +- std dev: [main] 145 ms +- 3 ms -> [main-again] 145 ms +- 3 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 145 ms +- 3 ms -> [shuffle-ceval-random-a] 144 ms +- 3 ms: 1.01x faster
Not significant!
Mean +- std dev: [main] 145 ms +- 3 ms -> [shuffle-ceval-random-b] 145 ms +- 3 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 145 ms +- 3 ms -> [shuffle-ceval-random-c] 144 ms +- 3 ms: 1.01x faster
Mean +- std dev: [main] 145 ms +- 3 ms -> [shuffle-ceval-random-d] 145 ms +- 3 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 145 ms +- 3 ms -> [shuffle-ceval-density] 145 ms +- 3 ms: 1.00x faster
Not significant!

sqlalchemy_imperative
=====================

Mean +- std dev: [main] 18.8 ms +- 0.3 ms -> [main-again] 18.8 ms +- 0.6 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 18.8 ms +- 0.3 ms -> [shuffle-ceval-random-a] 18.6 ms +- 0.2 ms: 1.01x faster
Mean +- std dev: [main] 18.8 ms +- 0.3 ms -> [shuffle-ceval-random-b] 18.8 ms +- 0.6 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 18.8 ms +- 0.3 ms -> [shuffle-ceval-random-c] 18.7 ms +- 0.5 ms: 1.01x faster
Mean +- std dev: [main] 18.8 ms +- 0.3 ms -> [shuffle-ceval-random-d] 18.8 ms +- 0.6 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 18.8 ms +- 0.3 ms -> [shuffle-ceval-density] 18.9 ms +- 0.8 ms: 1.00x slower
Not significant!

sqlite_synth
============

Mean +- std dev: [main] 2.76 us +- 0.06 us -> [main-again] 2.76 us +- 0.05 us: 1.00x faster
Not significant!
Mean +- std dev: [main] 2.76 us +- 0.06 us -> [shuffle-ceval-random-a] 2.73 us +- 0.04 us: 1.01x faster
Mean +- std dev: [main] 2.76 us +- 0.06 us -> [shuffle-ceval-random-b] 2.77 us +- 0.06 us: 1.00x slower
Not significant!
Mean +- std dev: [main] 2.76 us +- 0.06 us -> [shuffle-ceval-random-c] 2.77 us +- 0.05 us: 1.00x slower
Not significant!
Mean +- std dev: [main] 2.76 us +- 0.06 us -> [shuffle-ceval-random-d] 2.79 us +- 0.07 us: 1.01x slower
Mean +- std dev: [main] 2.76 us +- 0.06 us -> [shuffle-ceval-density] 2.75 us +- 0.06 us: 1.00x faster
Not significant!

sympy_expand
============

Mean +- std dev: [main] 515 ms +- 4 ms -> [main-again] 512 ms +- 6 ms: 1.01x faster
Mean +- std dev: [main] 515 ms +- 4 ms -> [shuffle-ceval-random-a] 512 ms +- 7 ms: 1.01x faster
Mean +- std dev: [main] 515 ms +- 4 ms -> [shuffle-ceval-random-b] 514 ms +- 2 ms: 1.00x faster
Mean +- std dev: [main] 515 ms +- 4 ms -> [shuffle-ceval-random-c] 512 ms +- 3 ms: 1.01x faster
Mean +- std dev: [main] 515 ms +- 4 ms -> [shuffle-ceval-random-d] 513 ms +- 5 ms: 1.00x faster
Mean +- std dev: [main] 515 ms +- 4 ms -> [shuffle-ceval-density] 512 ms +- 8 ms: 1.01x faster

sympy_integrate
===============

Mean +- std dev: [main] 22.4 ms +- 0.1 ms -> [main-again] 22.2 ms +- 0.1 ms: 1.01x faster
Mean +- std dev: [main] 22.4 ms +- 0.1 ms -> [shuffle-ceval-random-a] 22.2 ms +- 0.1 ms: 1.01x faster
Mean +- std dev: [main] 22.4 ms +- 0.1 ms -> [shuffle-ceval-random-b] 22.2 ms +- 0.1 ms: 1.01x faster
Mean +- std dev: [main] 22.4 ms +- 0.1 ms -> [shuffle-ceval-random-c] 22.2 ms +- 0.1 ms: 1.01x faster
Mean +- std dev: [main] 22.4 ms +- 0.1 ms -> [shuffle-ceval-random-d] 22.3 ms +- 0.1 ms: 1.00x faster
Mean +- std dev: [main] 22.4 ms +- 0.1 ms -> [shuffle-ceval-density] 22.3 ms +- 0.2 ms: 1.00x faster

sympy_sum
=========

Mean +- std dev: [main] 171 ms +- 1 ms -> [main-again] 171 ms +- 1 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 171 ms +- 1 ms -> [shuffle-ceval-random-a] 172 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 171 ms +- 1 ms -> [shuffle-ceval-random-b] 173 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 171 ms +- 1 ms -> [shuffle-ceval-random-c] 172 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 171 ms +- 1 ms -> [shuffle-ceval-random-d] 173 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 171 ms +- 1 ms -> [shuffle-ceval-density] 173 ms +- 3 ms: 1.01x slower

sympy_str
=========

Mean +- std dev: [main] 307 ms +- 4 ms -> [main-again] 306 ms +- 2 ms: 1.01x faster
Mean +- std dev: [main] 307 ms +- 4 ms -> [shuffle-ceval-random-a] 308 ms +- 2 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 307 ms +- 4 ms -> [shuffle-ceval-random-b] 309 ms +- 4 ms: 1.01x slower
Mean +- std dev: [main] 307 ms +- 4 ms -> [shuffle-ceval-random-c] 308 ms +- 6 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 307 ms +- 4 ms -> [shuffle-ceval-random-d] 308 ms +- 5 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 307 ms +- 4 ms -> [shuffle-ceval-density] 310 ms +- 6 ms: 1.01x slower

telco
=====

Mean +- std dev: [main] 6.46 ms +- 0.13 ms -> [main-again] 6.48 ms +- 0.14 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 6.46 ms +- 0.13 ms -> [shuffle-ceval-random-a] 6.48 ms +- 0.09 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 6.46 ms +- 0.13 ms -> [shuffle-ceval-random-b] 6.40 ms +- 0.12 ms: 1.01x faster
Mean +- std dev: [main] 6.46 ms +- 0.13 ms -> [shuffle-ceval-random-c] 6.49 ms +- 0.08 ms: 1.00x slower
Not significant!
Mean +- std dev: [main] 6.46 ms +- 0.13 ms -> [shuffle-ceval-random-d] 6.46 ms +- 0.11 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 6.46 ms +- 0.13 ms -> [shuffle-ceval-density] 6.48 ms +- 0.13 ms: 1.00x slower
Not significant!

tornado_http
============

Mean +- std dev: [main] 111 ms +- 3 ms -> [main-again] 110 ms +- 3 ms: 1.01x faster
Mean +- std dev: [main] 111 ms +- 3 ms -> [shuffle-ceval-random-a] 110 ms +- 3 ms: 1.01x faster
Not significant!
Mean +- std dev: [main] 111 ms +- 3 ms -> [shuffle-ceval-random-b] 111 ms +- 3 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 111 ms +- 3 ms -> [shuffle-ceval-random-c] 110 ms +- 3 ms: 1.01x faster
Not significant!
Mean +- std dev: [main] 111 ms +- 3 ms -> [shuffle-ceval-random-d] 111 ms +- 3 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 111 ms +- 3 ms -> [shuffle-ceval-density] 110 ms +- 3 ms: 1.00x faster
Not significant!

unpack_sequence
===============

Mean +- std dev: [main] 47.6 ns +- 1.8 ns -> [main-again] 46.1 ns +- 0.5 ns: 1.03x faster
Mean +- std dev: [main] 47.6 ns +- 1.8 ns -> [shuffle-ceval-random-a] 48.2 ns +- 2.1 ns: 1.01x slower
Not significant!
Mean +- std dev: [main] 47.6 ns +- 1.8 ns -> [shuffle-ceval-random-b] 48.5 ns +- 2.3 ns: 1.02x slower
Mean +- std dev: [main] 47.6 ns +- 1.8 ns -> [shuffle-ceval-random-c] 45.5 ns +- 2.2 ns: 1.05x faster
Mean +- std dev: [main] 47.6 ns +- 1.8 ns -> [shuffle-ceval-random-d] 45.5 ns +- 2.9 ns: 1.05x faster
Mean +- std dev: [main] 47.6 ns +- 1.8 ns -> [shuffle-ceval-density] 44.4 ns +- 2.4 ns: 1.07x faster

unpickle
========

Mean +- std dev: [main] 13.4 us +- 0.1 us -> [main-again] 14.0 us +- 0.8 us: 1.04x slower
Mean +- std dev: [main] 13.4 us +- 0.1 us -> [shuffle-ceval-random-a] 13.8 us +- 0.7 us: 1.03x slower
Mean +- std dev: [main] 13.4 us +- 0.1 us -> [shuffle-ceval-random-b] 13.8 us +- 0.3 us: 1.03x slower
Mean +- std dev: [main] 13.4 us +- 0.1 us -> [shuffle-ceval-random-c] 13.7 us +- 0.2 us: 1.02x slower
Mean +- std dev: [main] 13.4 us +- 0.1 us -> [shuffle-ceval-random-d] 13.9 us +- 0.2 us: 1.03x slower
Mean +- std dev: [main] 13.4 us +- 0.1 us -> [shuffle-ceval-density] 13.7 us +- 0.7 us: 1.02x slower

unpickle_list
=============

Mean +- std dev: [main] 4.98 us +- 0.06 us -> [main-again] 5.12 us +- 0.10 us: 1.03x slower
Mean +- std dev: [main] 4.98 us +- 0.06 us -> [shuffle-ceval-random-a] 5.03 us +- 0.09 us: 1.01x slower
Mean +- std dev: [main] 4.98 us +- 0.06 us -> [shuffle-ceval-random-b] 4.89 us +- 0.09 us: 1.02x faster
Mean +- std dev: [main] 4.98 us +- 0.06 us -> [shuffle-ceval-random-c] 5.09 us +- 0.27 us: 1.02x slower
Mean +- std dev: [main] 4.98 us +- 0.06 us -> [shuffle-ceval-random-d] 5.18 us +- 0.12 us: 1.04x slower
Mean +- std dev: [main] 4.98 us +- 0.06 us -> [shuffle-ceval-density] 4.97 us +- 0.08 us: 1.00x faster
Not significant!

unpickle_pure_python
====================

Mean +- std dev: [main] 276 us +- 2 us -> [main-again] 274 us +- 2 us: 1.01x faster
Mean +- std dev: [main] 276 us +- 2 us -> [shuffle-ceval-random-a] 277 us +- 3 us: 1.01x slower
Mean +- std dev: [main] 276 us +- 2 us -> [shuffle-ceval-random-b] 276 us +- 2 us: 1.00x slower
Not significant!
Mean +- std dev: [main] 276 us +- 2 us -> [shuffle-ceval-random-c] 272 us +- 2 us: 1.02x faster
Mean +- std dev: [main] 276 us +- 2 us -> [shuffle-ceval-random-d] 272 us +- 2 us: 1.01x faster
Mean +- std dev: [main] 276 us +- 2 us -> [shuffle-ceval-density] 277 us +- 4 us: 1.00x slower
Not significant!

xml_etree_parse
===============

Mean +- std dev: [main] 154 ms +- 3 ms -> [main-again] 156 ms +- 3 ms: 1.01x slower
Mean +- std dev: [main] 154 ms +- 3 ms -> [shuffle-ceval-random-a] 157 ms +- 6 ms: 1.02x slower
Mean +- std dev: [main] 154 ms +- 3 ms -> [shuffle-ceval-random-b] 156 ms +- 3 ms: 1.01x slower
Mean +- std dev: [main] 154 ms +- 3 ms -> [shuffle-ceval-random-c] 156 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 154 ms +- 3 ms -> [shuffle-ceval-random-d] 157 ms +- 5 ms: 1.02x slower
Mean +- std dev: [main] 154 ms +- 3 ms -> [shuffle-ceval-density] 158 ms +- 5 ms: 1.03x slower

xml_etree_iterparse
===================

Mean +- std dev: [main] 107 ms +- 2 ms -> [main-again] 107 ms +- 2 ms: 1.00x faster
Not significant!
Mean +- std dev: [main] 107 ms +- 2 ms -> [shuffle-ceval-random-a] 111 ms +- 3 ms: 1.04x slower
Mean +- std dev: [main] 107 ms +- 2 ms -> [shuffle-ceval-random-b] 112 ms +- 2 ms: 1.05x slower
Mean +- std dev: [main] 107 ms +- 2 ms -> [shuffle-ceval-random-c] 107 ms +- 2 ms: 1.01x slower
Mean +- std dev: [main] 107 ms +- 2 ms -> [shuffle-ceval-random-d] 109 ms +- 3 ms: 1.02x slower
Mean +- std dev: [main] 107 ms +- 2 ms -> [shuffle-ceval-density] 111 ms +- 2 ms: 1.04x slower

xml_etree_generate
==================

Mean +- std dev: [main] 83.9 ms +- 1.1 ms -> [main-again] 84.7 ms +- 0.9 ms: 1.01x slower
Mean +- std dev: [main] 83.9 ms +- 1.1 ms -> [shuffle-ceval-random-a] 84.9 ms +- 0.6 ms: 1.01x slower
Mean +- std dev: [main] 83.9 ms +- 1.1 ms -> [shuffle-ceval-random-b] 85.5 ms +- 1.0 ms: 1.02x slower
Mean +- std dev: [main] 83.9 ms +- 1.1 ms -> [shuffle-ceval-random-c] 84.6 ms +- 0.8 ms: 1.01x slower
Mean +- std dev: [main] 83.9 ms +- 1.1 ms -> [shuffle-ceval-random-d] 84.6 ms +- 0.9 ms: 1.01x slower
Mean +- std dev: [main] 83.9 ms +- 1.1 ms -> [shuffle-ceval-density] 84.2 ms +- 0.9 ms: 1.00x slower
Not significant!

xml_etree_process
=================

Mean +- std dev: [main] 61.7 ms +- 0.6 ms -> [main-again] 62.2 ms +- 0.5 ms: 1.01x slower
Mean +- std dev: [main] 61.7 ms +- 0.6 ms -> [shuffle-ceval-random-a] 62.1 ms +- 0.6 ms: 1.01x slower
Mean +- std dev: [main] 61.7 ms +- 0.6 ms -> [shuffle-ceval-random-b] 62.7 ms +- 0.8 ms: 1.02x slower
Mean +- std dev: [main] 61.7 ms +- 0.6 ms -> [shuffle-ceval-random-c] 62.1 ms +- 0.9 ms: 1.01x slower
Mean +- std dev: [main] 61.7 ms +- 0.6 ms -> [shuffle-ceval-random-d] 62.2 ms +- 0.8 ms: 1.01x slower
Mean +- std dev: [main] 61.7 ms +- 0.6 ms -> [shuffle-ceval-density] 61.9 ms +- 0.6 ms: 1.00x slower
Not significant!

Geometric mean
==============

main-again: 1.00x slower
shuffle-ceval-random-a: 1.01x slower
shuffle-ceval-random-b: 1.01x slower
shuffle-ceval-random-c: 1.00x faster
shuffle-ceval-random-d: 1.00x slower
shuffle-ceval-density: 1.00x slower

The noise seems much better now that we've properly isolated our benchmarking machine, but it'll never be perfect... we're still seeing individual results ranging from 11% slower to 7% faster on some runs.

pidigits stands out to me as particularly odd. Even though it looks to be basically just number-crunching and function calls, it experiences wide swings (even for the same commit) and never reports any deviation:

pidigits
========

Mean +- std dev: [main] 188 ms +- 0 ms -> [main-again] 208 ms +- 0 ms: 1.11x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-a] 189 ms +- 0 ms: 1.00x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-b] 209 ms +- 0 ms: 1.11x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-c] 188 ms +- 0 ms: 1.00x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-random-d] 188 ms +- 0 ms: 1.00x slower
Mean +- std dev: [main] 188 ms +- 0 ms -> [shuffle-ceval-density] 205 ms +- 0 ms: 1.09x slower

[gvanrossum]

Oh, this is interesting. We could compile a list of how big the variance is for each of the benchmarks over several randomized runs. That will give us a sense of how much to panic (or celebrate) if a certain benchmark suddenly sees a big swing.

That pidigits reports no stdev for a given build suggests that it really is about code alignment in the i-cache. It looks bimodal (listen to me using bug statistics words :-), either 1.00 or around 1.10. That could be due to either code alignment of the cases for the few opcodes it actually exercises, or due to the integer code being pushed to a slightly different boundary. But my vote's on alignment of the cases. So it would be useful to get a dynamic execution profile of the core loop.

[gvanrossum]

Looking at pidigits some more, the inner loop calls compose() and extract(), both of which use these opcodes a lot:

LOAD_FAST
UNPACK_SEQUENCE
STORE_FAST
BINARY_MULTIPLY

and these a little less:

BINARY_ADD
BINARY_FLOOR_DIVIDE
BUILD_TUPLE
RETURN_VALUE

So the spread of those opcodes (esp. the first four) across the i-cache might have a huge effect.

Not sure we let that guide us too much -- this is just an example of highly mathematical code, I'd rather optimize something like mypy. :-)

[brandtbucher]

Oh, this is interesting. We could compile a list of how big the variance is for each of the benchmarks over several randomized runs. That will give us a sense of how much to panic (or celebrate) if a certain benchmark suddenly sees a big swing.

Done! You can check out the results in #109.