Merge pull request #316 from perezjosibm/wip.fio_workloads

Update documentation. Minor reformating on the fio-parse-jsons.py

Author: perezjosibm

README.md

@@ -77,6 +77,7 @@ Optional tools and benchmarks can be used if desired:
  6. fio - benchmark suite with integrated posix, libaio, and librbd 
     support
  7. cosbench - object storage benchmark from Intel
+ 8. pytest - to run the unit tests.
 
 ## USER AND NODE SETUP
 
@@ -144,10 +145,12 @@ lab here:
 <https://github.com/ceph/cbt/blob/master/tools/mkpartmagna.sh>
 
 
-## CREATING A YAML FILE
+## CREATING A TEST PLAN YAML FILE
 
 CBT yaml files have a basic structure where you define a cluster and a set of
-benchmarks to run against it.  For example, the following yaml file creates a
+benchmarks to run against it.  The high level structure of a test plan is
+detailed in the [documentation](docs/TestPlanSchema.md).
+For example, the following yaml file creates a
 single node cluster on a node with hostname "burnupiX". A pool profile is
 defined for a 1x replication pool using 256 PGs, and that pool is used to run
 RBD performance tests using fio with the librbd engine.
@@ -250,6 +253,15 @@ called mkcephconf.py lets you automatically generate hundreds or thousands of
 ceph.conf files from defined ranges of different options that can then be used
 with cbt in this way.
 
+## RECENT FEATURES
+
+* Support for [workloads](docs/Workloads.md), that is sequence of performance tests, particularly
+useful to generate *Response latency curves*.
+
+* Automatic unit test [generation](docs/AutomaticUnitTestGeneration.md) for the Benchmark classes, intended to help
+refactoring to detect regressions.
+
+
 ## CONCLUSION
 
 There are many additional and powerful ways you can use cbt that are not yet

docs/AutomaticUnitTestGeneration.md

@@ -22,7 +22,7 @@ The following is an example of the execution of the script:
 ```
 An example of the expected normal ouput is shown below.
 
-![cbt_utests_gen](cbt_utest_gen.png)
+![cbt_utests_gen](./cbt_utests_gen.png)
 
 This would have created (or updated if existing already) the set of unit tests for the supported benchmarks.
 
@@ -35,7 +35,7 @@ The unit tests can be executed from the command line as follows:
 ```
 An example output showing a successful execution:
 
-![cbt_utests_gen](cbt_utest_gen.png)
+![cbt_utests_run](./cbt_utests_run.png)
 
 Note: the tests skipped above require an environment variable to be defined to identify the target nodes 
 for exercising pdsh.
@@ -69,7 +69,7 @@ whether some attributes has been changed, replaced or deleted. This is especiall
 during code refactoring.
 
 
-## Workflow recommeded
+## Workflow recommeded
 
 
 * Before starting a code refactoring effort, run the unit tests: they should all pass as shown above.

docs/TestPlanSchema.md

@@ -0,0 +1,70 @@
+# Test plan schema
+
+A valid test plan .yaml consists of the following compulsory sections at the top level (the level is 
+indicated by the indentation in .yaml: the top level has 0 indentation): 
+
+* `cluster` 
+* `benchmarks`.
+
+It may also have the following optional sections at the same level: 
+
+* `monitoring_profile`
+* `client_endpoints`.
+
+![top_level](./toplevel.png)
+
+## `cluster`
+
+The cluster section enumerates the components of the Ceph cluster relevant to CBT. There are two 
+general classes of components:
+
+* scalars: for example names whose value is a string, a numeric or a boolean;
+* collections: components that in turn contain further information, for example profile of pool 
+replication.
+
+The following are scalar compulsory entities:
+* a head node: this is a string indicating the node that starts the cluster.
+* a list of clients, each a string, representing a ssh-reachable host that has a benchmark 
+executable installed,
+* a list of osds nodes, each of which has at least a running OSD process.
+
+![cluster](./cluster.png)
+
+
+## `benchmarks`
+
+The benchmarks section consists of a non-empty list of collections, each describing a benchmark 
+entity.
+
+* A benchmark entity starts with its *name* (second level indentation), valid names are for example: 
+`radosbench`, `hsbench`, `kvmrbdfio`, `librbdfio`, etc.
+
+* The contents of the benchmark entity (third level indentation) consist of a collection of items 
+(either scalars or collections themselves).  Most of these entities represent options for the 
+command line invocation of the benchmark when executed by the clients.
+
+![benchmarks](./benchmarks.png)
+
+
+## `monitoring_profiles`
+
+
+The monitoring_profiles section consists of a non-empty list of of collections, each describing a 
+monitoring tool. 
+
+A monitoring entity starts with its name (at second level indentation). Currently supported are `perf`
+, `collectl`, `top`.
+
+The contents of the monitoring entity consists of :
+* a `nodes` (third level indentation) list of processes to monitor (by default the osd nodes), and 
+* an optional string `args` (third level indentation) to indicate the arguments to the monitoring tool.
+
+
+## `client_endpoints`
+
+The client_endpoints section consists of a non-empty list of collections, each associated to a 
+benchmark entity, and typically indicating  the driver for the benchmark. The client_endpoints, if 
+specified on a test plan, must be cross referenced by the benchmark section, and as such normally the 
+client_endpoints section precedes the benchmarks section in the test plan.
+
+See the dir `example/` for a number of test plan examples.

docs/Workloads.md

@@ -0,0 +1,46 @@
+# Workloads
+
+A workload is the specification of a sequence of tests to be executed in the order given.
+Typically this involves a *range* of values for a specific benchmark argument. The most used is
+the *queue depth*. Depending of the benchmark, this can be expressed as a function of the number
+of jobs (or threads, or processes), such that the increase number of these causes a proportional
+increase in the I/O. Specifiying workloads in this way permits to generate *response latency curves*
+from the results.
+
+The workload feature is currently supported for `librbdfio` only.
+
+![workloads](./workloads.png)
+
+* A `workloads`  section is composed by a non-empty collection. Each item in the workload has a free name,
+and contains in turn a collection of valid options with values for the benchmark.
+* For each of the `iodepth` and `numjobs` options, a range of integer values is permitted.
+
+During execution, any of the given values for the benchmark options in the global section are overwritten
+by the given values within the current test workload. The global values are restored once the workload test
+completes.
+
+As an example, the following specifies two workloads:
+
+* the first is named  `precondition` and consists of executing a random write over a queue depth of 4,
+(that is, the product of numjobs and iodepth), and indicates that monitoring should be disabled during the
+execution of the workload,
+* the second is named test1, and specifies a random read over the combinatorial of the provided sequences for
+the numjobs and iodepth, resp. That is, (1,1), (1,4), (1,8) .. (8,8).
+
+
+```yaml
+
+workloads:
+      precondition:
+        jobname: 'precond1rw'
+        mode: 'randwrite'
+        numjobs: [ 1 ]
+        iodepth: [ 4 ]
+        monitor: False # whether to run the monitors along the test
+      test1:
+        jobname: 'rr'
+        mode: 'randread'
+        numjobs: [ 1, 4, 8 ]
+        iodepth: [ 1, 4, 8 ]
+
+```