Add script to fetch benchmark results for execuTorch

[yangw-dev]

Summary

Provide methods and script to fetch all execuTorch benchamrk data from HUD API into two dataset,private and public, the script will:

• fetch all data from HUD API from input time range in UTC
• clean out records and tables with only FAILURE_REPORT due to job-level failures
• get all private table metrics, generate table_name and find intersected public table metrics
• generate private and public table groups
• output data

OutputType:

• run with excel-sheet export
• run with csv export
• run with dataframe format print
• run with json format print

See more guidance in README.md

the data is similar to the excel sheet generated manually in #10982
The result should be the same as the hud per model datatable:

helper methods: common.py

provide common.py helper method to convert back csv and excel sheets back to {"groupInfo":{}, "df":df.DataFrame} format.

run with

python3 .ci/scripts/benchmark_tooling/get_benchmark_analysis_data.py \
--startTime "2025-04-29T09:48:57" \
--endTime "2025-05-13T22:00:00" \
--outputType "excel" \
--models "mv3"

python3 .ci/scripts/benchmark_tooling/analyze_benchmark_stability.py \
--primary-file private.xlsx \
--reference-file public.xlsx

Generate excel files:
private.xlsx
public.xlsx

For instance you can find result for mv3 xnnq_q8 S22 Ultra android 14:

Latency Stability Analysis: table10 (Primary)
================================================================================
Model: mv3(xnnpack_q8)
Device: Samsung Galaxy S22 Ultra 5G (private)(Android 14)

Dataset Overview:
  - Number of samples: 88
  - Date range: 2025-04-29 09:48:57+00:00 to 2025-05-13 21:08:36+00:00

Central Tendency Metrics:
  - Mean latency: 2.91 ms
  - Median latency (P50): 2.54 ms
  - Mean trimmed latency: 2.41 ms
  - Median trimmed latency: 2.15 ms

Dispersion Metrics:
  - Standard deviation: 1.14 ms
  - Coefficient of variation (CV): 39.08%
  - Interquartile range (IQR): 0.82 ms
  - Trimmed standard deviation: 0.76 ms
  - Trimmed coefficient of variation: 31.60%

Percentile Metrics:
  - P50 (median): 2.54 ms
  - P90: 3.88 ms
  - P95: 4.60 ms
  - P99: 5.91 ms

Inter-Jitter Metrics (variability between runs):
  - Max/Min ratio: 5.6103
  - P99/P50 ratio: 2.3319
  - Mean rolling std (window=5): 0.79 ms

Intra-Jitter Metrics (variability within runs):
  - Mean trimming effect ratio: 15.37%
  - Max trimming effect ratio: 38.83%

Stability Assessment:
  - Overall stability score: 0.0/100
  - Overall stability rating: Poor

Interpretation:
  The benchmark shows poor stability (score: 0.0/100) with significant
  variation between runs (CV: 39.08%).
  Performance is unpredictable and may lead to inconsistent user experience.

  The significant difference between raw and trimmed means suggests
  considerable intra-run jitter (15.4%) with occasional outliers within benchmark runs.

  The max/min ratio of 5.61 indicates
  substantial performance differences between the best and worst runs.

  The P99/P50 ratio of 2.33 suggests
  occasional latency spikes that could affect tail latency sensitive applications.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11734

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job, 2 Pending, 3 Unrelated Failures

As of commit 1a3795e with merge base da36d8a ():

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / test-models-linux (add_mul, portable, linux.2xlarge) / linux-job (gh) (matched linux rule in flaky-rules.json)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
• pull / test-moshi-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• trunk / test-qnn-optimum-model (fp32, bert) / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[yangw-dev]

FYI, this method can be more general, but since only execuTorch is using it, i just make it execuTorch specific @huydhn

[yangw-dev]

the excel sheet has limit of sheet name len < 31, which can be easy to break in the future. @huydhn @guangy10 , I think instead of generate one file per category, maybe we can generate list of excel files stored in folders [private, public]

But right now with the hard-coded abbreviation, this works fine. THe excel sheet option is there in case people want to use it.

[huydhn]

Stamped to unblock! Let's start using the script and improve it along the way

[guangy10]

We configured a fixed list of matching names to list limited tables

I think we should make it more flexible as there are always new models, recipes, devices added. For example, we recently add more models (see on dash) from huggingface/optimum-executorch to the benchmark infra, and the list will keep expanding.

Similarly with new "devices" or "backends" available, we want to be able to query the results via the script as well.

the excel sheet has limit of sheet name len < 31, which can be easy to break in the future. @huydhn @guangy10 , I think instead of generate one file per category, maybe we can generate list of excel files stored in folders [private, public]

Yeah noticed the limits when I manually created the excel sheet. Ideally I'd like to get rid of the excel sheet by wiring the outputs from db to the analysis script directly. Given what is currently supported in this PR, what does the workflow look like if I want to rerun the analysis? That is, how is this script interfaced to the analysis script?

[guangy10]

Just to clarify, the whole purpose of this PR is to make it easier to analysis the stability of the benchmark by end of H1, and that should be part of the core of the benchmark infra. The work is incomplete if the newly added script can not work with the analysis script or require additional changes to be handled separately. The DevX is not getting better.

[yangw-dev]

Just to clarify, the whole purpose of this PR is to make it easier to analysis the stability of the benchmark by end of H1, and that should be part of the core of the benchmark infra. The work is incomplete if the newly added script can not work with the analysis script or require additional changes to be handled separately. The DevX is not getting better.

please review this again, this script is synced with analysis script with excel output now, test mv3 data results, and post the analysis example in the comment, for samples i saw, the result generated from the new script is similar to the results post in #10982

[guangy10]

Note that the examples names are still incorrect

[guangy10]

linter error to fix