Skip to content

Add script to fetch benchmark results for execuTorch #11734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Jun 23, 2025
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
3e8fa8f
final
yangw-dev Jun 16, 2025
07896ea
final
yangw-dev Jun 17, 2025
79f7788
final
yangw-dev Jun 17, 2025
101d631
final
yangw-dev Jun 17, 2025
83e76fe
final
yangw-dev Jun 17, 2025
3b4047f
final
yangw-dev Jun 17, 2025
1da3e87
final
yangw-dev Jun 17, 2025
2f604a0
final
yangw-dev Jun 17, 2025
9fa50a4
final
yangw-dev Jun 17, 2025
b56863d
final
yangw-dev Jun 17, 2025
b2ad5b6
final
yangw-dev Jun 17, 2025
4aced24
final
yangw-dev Jun 17, 2025
ab6e6cf
final
yangw-dev Jun 17, 2025
8e6956d
final
yangw-dev Jun 17, 2025
87ba460
final
yangw-dev Jun 17, 2025
5d22567
final
yangw-dev Jun 17, 2025
7a032e1
final
yangw-dev Jun 18, 2025
d33ce72
final
yangw-dev Jun 18, 2025
702478b
final
yangw-dev Jun 18, 2025
03cff40
final
yangw-dev Jun 18, 2025
44b792b
final
yangw-dev Jun 18, 2025
37965f3
final
yangw-dev Jun 18, 2025
908ee26
final
yangw-dev Jun 18, 2025
4d30776
final
yangw-dev Jun 18, 2025
e27779e
final
yangw-dev Jun 18, 2025
f8ba103
final
yangw-dev Jun 18, 2025
0fb1b2b
final
yangw-dev Jun 18, 2025
2878e96
Merge branch 'main' into addScript
yangw-dev Jun 18, 2025
04dbd97
final
yangw-dev Jun 18, 2025
95b30a4
final
yangw-dev Jun 18, 2025
d22af04
fix error test
yangw-dev Jun 18, 2025
25769da
fix error test
yangw-dev Jun 18, 2025
48d9d34
Merge branch 'main' into addScript
yangw-dev Jun 18, 2025
51b326a
fix error test
yangw-dev Jun 18, 2025
13261da
fix error test
yangw-dev Jun 18, 2025
ffe6839
Merge branch 'main' into addScript
yangw-dev Jun 18, 2025
d7a6652
fix error test
yangw-dev Jun 19, 2025
9182682
fix error test
yangw-dev Jun 19, 2025
33de04f
fix error test
yangw-dev Jun 19, 2025
68bf6f5
fix error test
yangw-dev Jun 19, 2025
3c2cbd2
fix error test
yangw-dev Jun 19, 2025
8de90bf
Merge branch 'main' into addScript
yangw-dev Jun 19, 2025
990ff44
fix error test
yangw-dev Jun 19, 2025
ed48f5f
Merge branch 'main' into addScript
yangw-dev Jun 19, 2025
99df0fe
fix error test
yangw-dev Jun 19, 2025
7be520e
setup link
yangw-dev Jun 20, 2025
600bb1a
setup link
yangw-dev Jun 20, 2025
4c0fdd2
setup link
yangw-dev Jun 21, 2025
faa2012
setup link
yangw-dev Jun 21, 2025
82c72eb
setup link
yangw-dev Jun 21, 2025
9e2ee88
setup link
yangw-dev Jun 21, 2025
e97850b
Merge branch 'main' into addScript
yangw-dev Jun 23, 2025
26dec1e
setup link
yangw-dev Jun 23, 2025
12b4ed7
Merge branch 'main' into addScript
yangw-dev Jun 23, 2025
1a3795e
setup link
yangw-dev Jun 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .ci/scripts/benchmark_tooling/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Benchmark Tooling

A library providing tools for benchmarking ExecutorchBenchmark data.

## Read Benchmark Data
`get_benchmark_analysis_data.py` fetches benchmark data from HUD Open API and processes it, grouping metrics by private and public devices.

### Quick Start

Install dependencies:
```bash
pip install -r requirements.txt
```

Run with default output (CLI):
```bash
python3 .ci/scripts/benchmark_tooling/get_benchmark_analysis_data.py --startTime "2025-06-11T00:00:00" --endTime "2025-06-17T18:00:00"
```

Additional options:
- `--silent`: Hide processing logs, show only results
- `--outputType df`: Display results in DataFrame format
- `--outputType excel --outputDir "{YOUR_LOCAL_DIRECTORY}"`: Generate Excel file with multiple sheets (`res_private.xlsx` and `res_public.xlsx`)
- `--outputType csv --outputDir "{YOUR_LOCAL_DIRECTORY}"`: Generate CSV files in folders (`private` and `public`)


### Python API Usage

To use the benchmark fetcher in your own scripts:

```python
import ExecutorchBenchmarkFetcher from benchmark_tooling.get_benchmark_analysis_data
fetcher = ExecutorchBenchmarkFetcher()
# Must call run first
fetcher.run()
private, public = fetcher.to_df()
```

## analyze_benchmark_stability.py
`analyze_benchmark_stability.py` analyzes the stability of benchmark data, comparing the results of private and public devices.

### Quick Start
Install dependencies:
```bash
pip install -r requirements.txt
```

```
python .ci/scripts/benchmark_tooling/analyze_benchmark_stability.py \
Benchmark\ Dataset\ with\ Private\ AWS\ Devices.xlsx \
--reference_file Benchmark\ Dataset\ with\ Public\ AWS\ Devices.xlsx
```
Empty file.
Loading
Loading