v0.8.0: add a confidence metric and filter

- how much of a clear winner is the attack time the algorithm chooses? or would it be hard to pick out the peak because there's high response in the neighborhood? (say, a ringing attack, or a two-part attack like a clap sound)
- is it far enough away from the chosen attack time that it would actually impact the sync?
based on all that, I designed a goofy lil confidence metric with some fun math, and it seems to do a reasonable job at matching my own visual inspections

- also there's now an option of only unbiasing charts/files above a certain confidence. this is helpful I swear

Author: telperion

README.md

@@ -103,6 +103,30 @@ This is the most informative plot (imo), and can also help identify other sync i
 ![Convolution response of Perfect (ITG1)](doc/bias-postkernel-Perfect.png)
 
 
+## Confidence
+
+Not every track has a sharply defined beat throughout. Sometimes the sync fingerprint can pinpoint the attack clearly; other tracks might exhibit a lot of uncertainty, or the simfile skeleton might not define the correct BPMs. This tool is (for the moment) only interested in offset identification and adjustment, and we don't want to mess with files that are unclear - or in a state where moving the offset won't make the sync better. With that in mind, a **confidence metric** is introduced.
+
+### What makes a good confidence metric?
+What could cause the algorithm to pick an incorrect sync bias? Let's consider the following:
+- How much of a clear winner is the attack time the algorithm chooses? Or, would it be hard to pick out the peak because there's high response in the neighborhood? (say, a ringing attack, or a two-part attack like a clap sound)
+- Is this extra algorithmic response far enough away from the chosen attack time that it would actually impact the sync?
+
+With that in mind, the following calculations are performed:
+1. For each point in the flattened convolution response (the white squiggle in the sync fingerprint plots), measure the following:
+    - *v*, the relative distance from the response's median, compared to the identified peak.
+    - *d*, the time difference from the identified peak.
+1. Balance these two measurements using power functions and multiply them together to calculate the "unconfidence" this point contributes. (The current confidence calculation uses *v*^4 × *d*^1.5.)
+1. Sum all of these "unconfidence" values and subtract from 1 to obtain a "confidence" value.
+1. Apply some perceptual scaling and express as a percentage.
+
+The actual values returned from the confidence metric don't have an intrinsic meaning - that is, there's nothing in the plot you can point to that directly produces it - but it's expected that "messier" plots result in lower confidence, and "sharper" plots in higher confidence.
+
+Note that a value of 100% or near-100% confidence does not mean the current sync is *correct*, just that the algorithm can't see anything in the rest of the fingerprint to convince it that the peak could possibly lie elsewhere.
+
+The GUI includes a control to tune the minimum confidence to apply unbiasing at; it's expressed in percentage out of 100%. The CLI also offers this parameter, but in terms of proportion out of unity - for example, if the user wants to only apply unbiasing over 80% confidence, they would pass `--confidence 0.80` at the command line invocation. The CSV output also expresses the confidence as a proportion out of unity.
+
+
 ## Future plans
 - Code cleanup
 - Performance optimization (need to move to MVC model :weary:)

nine-or-null/nine_or_null.ipynb

@@ -253,6 +253,143 @@
     "    if os.path.isdir(os.path.join(pack_dir, d)):\n",
     "        print(d)\n"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "a = 3.14159265\n",
+    "d = f'{a:0.3f}'\n",
+    "print(d)\n",
+    "b = '{:0.3f}'\n",
+    "c = b.format(a)\n",
+    "print(c)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for test_simfile_path in [\n",
+    "    r'C:\\Games\\ITGmania\\Songs\\ephemera v0.2\\crew -All Hands on the Deck-\\crew.ssc',\n",
+    "    r'C:\\Games\\ITGmania\\Songs\\ephemera v0.2\\PPBQ\\ppbq.ssc',\n",
+    "    r'C:\\Games\\ITGmania\\Songs\\ephemera v0.2\\Adrenalina\\adrenalina.ssc'\n",
+    "]:\n",
+    "    base_simfile = simfile.open(test_simfile_path)\n",
+    "    for chart_index, chart in enumerate(base_simfile.charts):\n",
+    "        if any(k in chart for k in ['OFFSET', 'BPMS', 'STOPS', 'DELAYS', 'WARPS']):\n",
+    "            print(f'{base_simfile.title}: {chart_index} ({chart.difficulty}) has split timing')\n",
+    "\n",
+    "    # for k, v in base_simfile.charts[1].items():\n",
+    "    #     print(f'{k}: {v}')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "a = [None, 3]\n",
+    "print(os.path.join(os.getcwd(), '*'))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Two aspects to confidence:\n",
+    "# - is the overall max response a clear winner, or are there other contenders?\n",
+    "#     - (second-highest - median) / (highest - median)\n",
+    "# - is the response outside of the max tight, or does it have a lot of variance/noise?\n",
+    "#     - stdev after scaling\n",
+    "\n",
+    "import os\n",
+    "import csv\n",
+    "import glob\n",
+    "import numpy as np\n",
+    "from matplotlib import pyplot as plt\n",
+    "\n",
+    "edge_discard = 3\n",
+    "\n",
+    "conv_list = glob.glob(r'C:\\Games\\ITGmania\\Songs\\ITL Online 2023\\__bias-check\\convolution-*.csv')\n",
+    "\n",
+    "conv_data = {}\n",
+    "conv_results = []\n",
+    "\n",
+    "for i, f in enumerate(conv_list):\n",
+    "    t = []\n",
+    "    v = []\n",
+    "    with open(f, 'r', encoding='ascii') as fp:\n",
+    "        reader = csv.reader(fp)\n",
+    "        for row in reader:\n",
+    "            t += [float(row[0])]\n",
+    "            v += [float(row[1])]\n",
+    "    v_clip = v[edge_discard:-edge_discard]\n",
+    "    v_clip = np.interp(v_clip, (min(v_clip), max(v_clip)), (0, 1))\n",
+    "    t_clip = np.array(t[edge_discard:-edge_discard])\n",
+    "    v_std = np.std(v_clip)\n",
+    "    v_mean = np.mean(v_clip)\n",
+    "    v_median = np.median(v_clip)\n",
+    "    v_argmax = np.argmax(v_clip)\n",
+    "    v_max = v_clip[v_argmax]\n",
+    "    v_20 = np.percentile(v_clip, 20)\n",
+    "    v_80 = np.percentile(v_clip, 80)\n",
+    "\n",
+    "    # Local maxima\n",
+    "    v_moving_diff = v_clip[1:] - v_clip[:-1]\n",
+    "    v_local_maxima = [i+1 for i, vmd in enumerate(zip(v_clip[:-2], v_clip[1:-1], v_clip[2:])) if (vmd[1] > vmd[0]) and (vmd[1] > vmd[2])]\n",
+    "    v_peaks = sorted(zip(v_local_maxima, v_clip[v_local_maxima]), key=lambda x: x[1], reverse=True)[:6]\n",
+    "    # print([f'{v[0]}: {v[1]}' for v in v_peaks])\n",
+    "    N_SAMPLES_NOT_NEAR = 10\n",
+    "    v_peaks_not_near = [v for v in v_peaks if abs(v[0] - v_peaks[0][0]) > N_SAMPLES_NOT_NEAR]\n",
+    "    maxness = 0\n",
+    "    if len(v_peaks_not_near) > 0:\n",
+    "        maxness = (v_peaks_not_near[0][1] - v_median) / (v_peaks[0][1] - v_median)\n",
+    "\n",
+    "    # Another approach...\n",
+    "    THEORETICAL_UPPER = 0.83\n",
+    "    NEARNESS_SCALAR = 10    # milliseconds\n",
+    "    NEARNESS_OFFSET = 0.5   # milliseconds\n",
+    "    \n",
+    "    v_max_check = np.vstack((np.zeros_like(v_clip), (v_clip - v_median) / (v_max - v_median)))\n",
+    "    v_max_rivaling = np.max(v_max_check, axis=0)\n",
+    "    t_close_check = np.vstack((np.zeros_like(t_clip), abs(t_clip - t_clip[v_argmax]) - NEARNESS_OFFSET)) / NEARNESS_SCALAR\n",
+    "    t_close_enough = np.max(t_close_check, axis=0)\n",
+    "    max_influence = np.power(v_max_rivaling, 4) * np.power(t_close_enough, 1.5)\n",
+    "    total_max_influence = np.sum(max_influence) / np.size(max_influence)\n",
+    "    confidence = min(1, (1 - np.power(total_max_influence, 0.2)) / THEORETICAL_UPPER)\n",
+    "\n",
+    "    print(f'{i:3d}/{len(conv_list):3d} {os.path.split(f)[1]:50s}: median = {v_median:0.6f}, stdev = {v_std:0.6f}, confidence = {confidence*100:0.2f}%')\n",
+    "    conv_results.append((os.path.split(f)[1], confidence))\n",
+    "\n",
+    "    if confidence < 0.75 or i % 20 == 0:\n",
+    "        plt.figure(figsize=(6, 6))\n",
+    "        plt.title(os.path.split(f)[1] + f'\\nstdev = {v_std:0.6f}, iqr = {v_80-v_20:0.6f}, confidence = {confidence*100:0.2f}%')\n",
+    "        # plt.plot(t[edge_discard:-edge_discard], v[edge_discard:-edge_discard])\n",
+    "        # plt.plot(sorted(v_clip))\n",
+    "        plt.plot(v_clip)\n",
+    "        plt.plot(max_influence, 'm')\n",
+    "        #plt.plot(v_max_rivaling, 'c')\n",
+    "        # plt.plot(np.full_like(v_clip, v_mean + v_std*3), 'r')\n",
+    "        # plt.plot(np.full_like(v_clip, v_mean),           'g')\n",
+    "        # plt.plot(np.full_like(v_clip, v_mean - v_std*3), 'r')\n",
+    "        plt.plot(np.full_like(v_clip, v_20),     'c')\n",
+    "        plt.plot(np.full_like(v_clip, v_median), 'g')\n",
+    "        plt.plot(np.full_like(v_clip, v_80),     'c')\n",
+    "        #plt.plot([v[0] for v in v_peaks_not_near], [v[1] for v in v_peaks_not_near], 'k.')\n",
+    "        plt.savefig(\"conf-\" + os.path.split(f)[1][12:-4] + \".png\")\n",
+    "        plt.close()\n",
+    "\n",
+    "for v in sorted(conv_results, key=lambda v: v[1]):\n",
+    "    print(f'{v[0]:50s}: {v[1]*100:0.2f}% confidence')\n"
+   ]
   }
  ],
  "metadata": {

nine-or-null/nine_or_null/__init__.py

@@ -1,4 +1,4 @@
-_VERSION = '0.7.1'
+_VERSION = '0.8.0'
 
 from collections.abc import Container
 import csv
@@ -25,6 +25,9 @@
     'path',
     'slot',
     'bias',
+    'conf',
+    'interquintile',
+    'stdev',
     'paradigm',
     'timestamp',
     'fingerprint_ms',
@@ -47,6 +50,7 @@
     'consider_null':    'Consider charts close enough to 0ms bias to be "correct" under the null (StepMania) sync paradigm.',
     'consider_p9ms':    'Consider charts close enough to +9ms bias to be "correct" under the In The Groove sync paradigm.',
     'tolerance':        'If a simfile\'s sync bias lands within a paradigm ± this tolerance, that counts as "close enough".',
+    'confidence_limit': 'If the confidence in a simfile\'s sync bias is below this value, it will not be considered for unbiasing.',
     'fingerprint_ms':   '[ms] Time margin on either side of the beat to analyze.',
     'window_ms':        '[ms] The spectrogram algorithm\'s moving window parameter.',
     'step_ms':          '[ms] Controls the spectrogram algorithm\'s overlap parameter, but expressed as a step size.',
@@ -56,6 +60,9 @@
     'full_spectrogram': 'Analyze the full spectrogram in one go - this will make the program run slower...',
     'to_paradigm':      'Choose a target paradigm for the pack unbiasing step. This will modify your simfiles!'
 }
+_THEORETICAL_UPPER = 0.83
+_NEARNESS_SCALAR = 10    # milliseconds
+_NEARNESS_OFFSET = 0.5   # milliseconds
 
 class FloatRange(Container):
     # Endpoint inclusive.
@@ -89,6 +96,7 @@ class KernelTarget(IntEnum):
     'consider_null':    True,
     'consider_p9ms':    True,
     'tolerance':        3.0,
+    'confidence_limit': 80,
     'fingerprint_ms':   50,
     'window_ms':        10,
     'step_ms':          0.2,
@@ -534,10 +542,32 @@ def check_sync_bias(simfile_dir, base_simfile, chart_index=None, report_path=Non
     fingerprint_times_ms = fingerprint_times * 1e3
 
     # Choose the highest response to the convolution as the downbeat attack
-    sync_bias_ms = fingerprint_times_ms[np.argmax(post_kernel_flat[edge_discard:-edge_discard]) + edge_discard] + magic_offset_ms
+    post_kernel_clip = post_kernel_flat[edge_discard:-edge_discard]
+    i_max = np.argmax(post_kernel_clip)
+    sync_bias_ms = fingerprint_times_ms[i_max + edge_discard] + magic_offset_ms
     probable_bias = guess_paradigm(sync_bias_ms, short_paradigm=False, **kwargs)
     # print(f'Sync bias: {sync_bias:0.3f} ({probable_bias})')
 
+    # Calculate a confidence statistic based on the presence of conflicting
+    # high-level response distant from the chosen peak
+    v_clip = np.interp(post_kernel_clip, (min(post_kernel_clip), max(post_kernel_clip)), (0, 1))
+    t_clip = fingerprint_times_ms[edge_discard:-edge_discard]
+    v_std = np.std(v_clip)
+    v_mean = np.mean(v_clip)
+    v_median = np.median(v_clip)
+    v_20 = np.percentile(v_clip, 20)
+    v_80 = np.percentile(v_clip, 80)
+    v_max = v_clip[i_max]
+    v_max_check = np.vstack((np.zeros_like(v_clip), (v_clip - v_median) / (v_max - v_median)))
+    v_max_rivaling = np.max(v_max_check, axis=0)
+    t_close_check = np.vstack((np.zeros_like(t_clip), abs(t_clip - t_clip[i_max]) - _NEARNESS_OFFSET)) / _NEARNESS_SCALAR
+    t_close_enough = np.max(t_close_check, axis=0)
+    max_influence = np.power(v_max_rivaling, 4) * np.power(t_close_enough, 1.5)
+    total_max_influence = np.sum(max_influence) / np.size(max_influence)
+    sync_confidence = min(1, (1 - np.power(total_max_influence, 0.2)) / _THEORETICAL_UPPER)
+    conv_interquintile = v_80 - v_20
+    conv_stdev = v_std
+
     full_title = get_full_title(base_simfile)
 
     plot_tag_vars = kwargs.get('tag_vars', {}) 
@@ -553,16 +583,21 @@ def check_sync_bias(simfile_dir, base_simfile, chart_index=None, report_path=Non
         fingerprint['steps_type'] = chart['STEPSTYPE']
         fingerprint['chart_slot'] = chart['DIFFICULTY']
         chart_tag = ' ' + slot_abbreviation(chart['STEPSTYPE'], chart['DIFFICULTY'], chart_index=chart_index, paradigm=guess_paradigm(sync_bias_ms, **kwargs))
-    fingerprint['sample_rate'] = audio.frame_rate
-    fingerprint['beat_digest'] = digest
+    fingerprint['sample_rate']  = audio.frame_rate
+    fingerprint['beat_digest']  = digest
     fingerprint['beat_indices'] = np.array(beat_indices)
-    fingerprint['freq_domain'] = acc
-    fingerprint['post_kernel'] = post_kernel
-    fingerprint['convolution'] = post_kernel_flat
-    fingerprint['frequencies'] = frequencies * 1e-3
-    fingerprint['time_values'] = fingerprint_times_ms
-    fingerprint['bias_result'] = sync_bias_ms
-    fingerprint['plots_title'] = f'Sync fingerprint{plot_tag}\n{simfile_artist} - "{full_title}"{chart_tag}\nSync bias: {sync_bias_ms:+0.1f} ms ({probable_bias})'
+    fingerprint['freq_domain']  = acc
+    fingerprint['post_kernel']  = post_kernel
+    fingerprint['convolution']  = post_kernel_flat
+    fingerprint['frequencies']  = frequencies * 1e-3
+    fingerprint['time_values']  = fingerprint_times_ms
+    fingerprint['bias_result']  = sync_bias_ms
+    fingerprint['confidence']   = sync_confidence
+    fingerprint['conv_stdev']   = conv_stdev
+    fingerprint['conv_quint']   = conv_interquintile
+    fingerprint['plots_title']  = \
+        f'Sync fingerprint{plot_tag}\n{simfile_artist} - "{full_title}"{chart_tag}' + \
+        f'\n{sync_bias_ms:+0.1f} ms bias ({probable_bias}), {round(sync_confidence*100):d}% conf'
 
     sanitized_title = slugify(full_title + chart_tag, allow_unicode=False)
     target_axes = []
@@ -574,6 +609,12 @@ def check_sync_bias(simfile_dir, base_simfile, chart_index=None, report_path=Non
 
     plot_fingerprint(fingerprint, target_axes, **kwargs)
 
+    # DEBUG: convolution output for confidence research
+    with open(os.path.join(report_path, f'convolution-{sanitized_title}.csv'), 'w', newline='', encoding='ascii') as conv_fp:
+        writer = csv.writer(conv_fp)
+        for t, v in zip(fingerprint_times_ms, post_kernel_flat):
+            writer.writerow([f'{t:0.6f}', f'{v:0.6f}'])
+
     for i, v in enumerate(['freqdomain', 'beatdigest', 'postkernel']):
         fig = target_figs[i]
         if show_intermediate_plots:
@@ -696,6 +737,9 @@ def batch_process(root_path=None, **kwargs):
             for split_chart in charts_within:
                 fp = check_sync_bias(p, base_simfile, chart_index=split_chart, save_plots=True, show_intermediate_plots=False, **kwargs)
                 sync_bias_ms = fp['bias_result']
+                sync_confidence = fp['confidence']
+                conv_quint = 'conv_quint' in fp and f"{fp['conv_quint']:0.6f}" or '----'
+                conv_stdev = 'conv_stdev' in fp and f"{fp['conv_stdev']:0.6f}" or '----'
 
                 chart_abbr = '*'
                 if split_chart is not None:
@@ -707,14 +751,16 @@ def batch_process(root_path=None, **kwargs):
 
                 logging.info(f'\t{fp_lookup}')
                 logging.info(f'\tderived sync bias = {sync_bias_ms:+0.1f} ms ({guess_paradigm(sync_bias_ms, short_paradigm=False, **kwargs)})')
+                logging.info(f'\tbias confidence = {round(sync_confidence*100):3d}% (interquintile spread = {conv_quint}, stdev = {conv_stdev})')
                 if gui_hook is not None:
                     row_index = len(fingerprints)-1
                     gui_hook.grid_results.InsertRows(row_index, 1)
                     gui_hook.grid_results.SetCellValue(row_index, 0, os.path.relpath(p, root_path))
                     gui_hook.grid_results.SetCellValue(row_index, 1, chart_abbr)
                     gui_hook.grid_results.SetCellValue(row_index, 2, f'{sync_bias_ms:+0.1f}')
-                    gui_hook.grid_results.SetCellValue(row_index, 3, guess_paradigm(sync_bias_ms, **kwargs))
-                    gui_hook.grid_results.MakeCellVisible(row_index, 3)
+                    gui_hook.grid_results.SetCellValue(row_index, 3, f'{round(sync_confidence*100):3d}%')
+                    gui_hook.grid_results.SetCellValue(row_index, 4, guess_paradigm(sync_bias_ms, **kwargs))
+                    gui_hook.grid_results.MakeCellVisible(row_index, 4)
                     for j in range(4):
                         gui_hook.grid_results.SetReadOnly(row_index, j)
                     gui_hook.grid_results.ForceRefresh()
@@ -724,6 +770,9 @@ def batch_process(root_path=None, **kwargs):
                         'path': os.path.relpath(p, root_path),
                         'slot': chart_abbr,
                         'bias': f'{sync_bias_ms:0.3f}',
+                        'conf': f'{sync_confidence:0.4f}',
+                        'interquintile': f"{fp.get('conv_quint', None)}",
+                        'stdev': f"{fp.get('conv_stdev', None)}",
                         'paradigm': guess_paradigm(sync_bias_ms, **kwargs),
                         'timestamp': timestamp(),
                         'sample_rate': fp.get('sample_rate', None)
@@ -755,7 +804,8 @@ def batch_adjust(fingerprints, target_bias, **params):
         if affect_rows is not None and i not in affect_rows:
             continue
         current_paradigm = fingerprints[k].get('bias_adjust', guess_paradigm(fingerprints[k]['bias_result'], **params))
-        if current_paradigm == source_bias:
+        current_confidence = fingerprints[k].get('confidence', 100)
+        if current_paradigm == source_bias and current_confidence >= params.get('confidence_limit', 0):
             logging.info(f'\t{k}')
             # Open simfile
             p, abbr = os.path.split(k)
@@ -782,7 +832,7 @@ def batch_adjust(fingerprints, target_bias, **params):
                         steps_type, chart_slot, chart_index = slot_expansion(abbr)
                         if chart_index is None:
                             chart_index = [i for i, c in enumerate(sm.charts) if c['STEPSTYPE'] == steps_type and c['DIFFICULTY'] == chart_slot][0]
-                        prev_offset = float(sm.charts[chart_index]['OFFSET'])
+                        prev_offset = float(sm.charts[chart_index].get('OFFSET', sm.offset))
                         new_offset = prev_offset + bias_shift
                         logging.info(f'\t{prev_offset:6.3f} -> {new_offset:6.3f}: {k}')
                         sm.charts[chart_index]['OFFSET'] = f'{new_offset:0.3f}'
@@ -793,7 +843,7 @@ def batch_adjust(fingerprints, target_bias, **params):
                     if gui_hook is not None:
                         font_cell = gui_hook.grid_results.GetCellFont(i, 0)
                         gui_hook.grid_results.SetCellValue(i, 2, f"{fingerprints[k]['bias_result']:+0.1f}")
-                        gui_hook.grid_results.SetCellValue(i, 3, target_bias)
+                        gui_hook.grid_results.SetCellValue(i, 4, target_bias)
                         for j in range(gui_hook.grid_results.GetNumberCols()):
                             gui_hook.grid_results.SetCellFont(i, j, font_cell.MakeBold())