Document, consolidate and complement the execution flow

[jest]

Recently there has been some changes made in hyperfine that affect how and when different commands are run. To have a common understanding, I think it is a good idea to summarize the current state.

Also, as mentioned in #219, there are cases when running preparation command only once per benchmarked command is desirable. In analysis below I show it is not so hard to do and does not break the current flow of execution.

Current model

Hyperfine gives us the possibility to run commands multiple times. Let's analyze the command line and name some workflow-related concepts:

hyperfine [OPTIONS] <command>...

• command --- a single tested command; equal to one of <command>-s, or --- when using parameters --- generated from one of <command> templates by applying one of parameter values
• measured run --- a single, measured execution of one of commands; affecting options are --min-runs, --max-runs, --runs
• warmup run --- non-measured execution of a command, used e.g. to remove the bias related to running commands for the first time. Affecting options: --warmup
• preparation --- an initializing code, run before each measured and warmup run[1] of a single command. The preparation can be the same for all commands, or specific to each of commands. Affecting options: --prepare
• cleanup --- a cleanup code, run once after all warmup and measured runs of a single command. The cleanup is specified the same for all commands, but is subject to parameter substitution. Affecting options: --cleanup
• parameters --- a name and a list of values --- either explicit one, or generated --- used to easily create a number of commands, one for each value from a list. There can be only one list of values. If multiple <command>...-s are specified, each of them is treated as a template for each value from the the parameters list. Affecting options: --parameter-scan, --parameter-step-size, --parameter-list

[1]: since version 1.9; previously preparation and cleanup was not run before warmups; see #228

To illustrate the concepts, running such a code:

hyperfine --show-output --warmup 2 --runs 2 \
  --parameter-scan n 1 2 \
  --prepare 'echo "# prep-A {n}"' --prepare 'echo "# prep-B {n}"' \
  --prepare 'echo "# prep-A {n}"' --prepare 'echo "# prep-B {n}"' \
  --cleanup 'echo "#---- clean {n}"' \
  'echo "#-- cmd-A {n}"' 'echo "#-- cmd-B {n}"'

we receive such an output (ignoring all hyperfine output, beautifying with newlines between different commands/parameters and marking what is really benchmarked with !!!):

# prep-A 1
#-- cmd-A 1
# prep-A 1
#-- cmd-A 1
# prep-A 1
#-- cmd-A 1      !!!
# prep-A 1
#-- cmd-A 1      !!!
#---- clean 1

# prep-B 1
#-- cmd-B 1
# prep-B 1
#-- cmd-B 1
# prep-B 1
#-- cmd-B 1      !!!
# prep-B 1
#-- cmd-B 1      !!!
#---- clean 1

# prep-A 2
#-- cmd-A 2
# prep-A 2
#-- cmd-A 2
# prep-A 2
#-- cmd-A 2      !!!
# prep-A 2
#-- cmd-A 2      !!!
#---- clean 2

# prep-B 2
#-- cmd-B 2
# prep-B 2
#-- cmd-B 2
# prep-B 2
#-- cmd-B 2      !!!
# prep-B 2
#-- cmd-B 2      !!!
#---- clean 2

Analysis and proposed workflow

Let's call each segment separated with newlines in the above output as a batch . A batch corresponds to running one command, generated from one of the <command>... "templates" given on the command line, with possibly {var} substituted for one parameter value. The batch would be ideally run with all preparations and cleanup.

I think we can agree that warmup runs and measured runs should be exactly the same. So we do not differentiate the prepare commands for those runs in a single batch.

The comprehensive workflow for a single batch could look like this:

|-- batch-prepare-cmd

( ... warmup phase ... )

|-- prepare
|-- cmd
|-- prepare
|-- cmd
|-- ... (repeat --warmup times)

( ... end of warmup phase ... )

|-- warmups-cleanup-cmd

( ... measured phase ... )

|-- prepare
|-- cmd
|-- prepare
|-- cmd
|-- ... (repeat required number of times)

( ... end of measured phase ... )

|-- batch-cleanup-cmd

The proposed mapping of above commands to existing functionality:

• batch-prepare-cmd -- missing, need to be added
• prepare -- one of --prepare with {var} substitution applied, already implemented
• batch-cleanup-cmd -- --cleanup with {var} substitution applied -- already implemented

So looks like we are only missing a command to prepare a batch.

Let me know if I miss some important use case in the above workflow.

Additionally, it would be nice to have multiple --cleanup-s, as we have multiple --prepare-s.

[sharkdp]

Thank you very much for your detailed analysis. I am planning to take a look soon.

[sharkdp]

I think what would be really helpful: a visual execution flow graph, maybe done in graphviz (or similar) to make it easy to edit in the future.

[sharkdp]

@jest Would you mind having a look at the following diagram? This documents the current execution order. It also includes the new --setup option. Would something like this work for you (as documentation)?

[jest]

Looks great, esp. color-coded!

Maybe the only think, totally non-crucial, I could somehow mention in addition is the parameter substitution possibility in setup and prepare; they look like are configured global, but can be tweaked to match the command run.

Great work!

[sharkdp]

Good point. I think this should probably be in a separate chapter of the (not yet existing) documentation though.