test(ab): allow ignoring specific metric with ignore list

Currently, if something matches the A/B-testing ignore list, then all
metrics emitted from a test with a dimension set that is a super set of
an ignored one is ignored. Refine this to allow only ignoring specific
metrics.

Realize this by synthesizing a fake dimension called 'metric' that
stores the metric.

This will later be used when we introduce block latency tests, as we
will want to A/B-test throughput but ignore latency in scenarios where
fio's async workload generator is used.

Signed-off-by: Patrick Roy <roypat@amazon.co.uk>

Author: roypat

tools/ab_test.py

@@ -50,7 +50,7 @@
 
 
 def is_ignored(dimensions) -> bool:
-    """Checks whether the given dimensions match a entry in the IGNORED dictionary above"""
+    """Checks whether the given dimensions match an entry in the IGNORED dictionary above"""
     for high_variance in IGNORED:
         matching = {key: dimensions[key] for key in high_variance if key in dimensions}
 
@@ -258,7 +258,7 @@ def analyze_data(
 
     failures = []
     for (dimension_set, metric), (result, unit) in results.items():
-        if is_ignored(dict(dimension_set)):
+        if is_ignored(dict(dimension_set) | {"metric": metric}):
             continue
 
         print(f"Doing A/B-test for dimensions {dimension_set} and property {metric}")