Skip to content

Commit 3a0ad10

Browse files
gateoverflow-admingateoverflow-admin
committed
Updated results summary
1 parent bce112b commit 3a0ad10

File tree

5 files changed

+16
-16
lines changed

5 files changed

+16
-16
lines changed

‎README.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
11
Please download [summary.xlsx](summary.xlsx) to view the most recent results.
22
```
3-
[2024-10-14 13:55:53,434 submission_checker1.py:2936 INFO] Results=22, NoResults=0, Power Results=0
4-
[2024-10-14 13:55:53,434 submission_checker1.py:2943 INFO] ---
5-
[2024-10-14 13:55:53,434 submission_checker1.py:2944 INFO] Closed Results=0, Closed Power Results=0
3+
[2024-10-14 13:56:43,464 submission_checker1.py:2936 INFO] Results=22, NoResults=0, Power Results=0
4+
[2024-10-14 13:56:43,464 submission_checker1.py:2943 INFO] ---
5+
[2024-10-14 13:56:43,464 submission_checker1.py:2944 INFO] Closed Results=0, Closed Power Results=0
66
7-
[2024-10-14 13:55:53,434 submission_checker1.py:2949 INFO] Open Results=22, Open Power Results=0
7+
[2024-10-14 13:56:43,464 submission_checker1.py:2949 INFO] Open Results=22, Open Power Results=0
88
9-
[2024-10-14 13:55:53,434 submission_checker1.py:2954 INFO] Network Results=0, Network Power Results=0
9+
[2024-10-14 13:56:43,464 submission_checker1.py:2954 INFO] Network Results=0, Network Power Results=0
1010
11-
[2024-10-14 13:55:53,434 submission_checker1.py:2959 INFO] ---
12-
[2024-10-14 13:55:53,434 submission_checker1.py:2961 INFO] Systems=18, Power Systems=0
13-
[2024-10-14 13:55:53,434 submission_checker1.py:2962 INFO] Closed Systems=0, Closed Power Systems=0
14-
[2024-10-14 13:55:53,434 submission_checker1.py:2967 INFO] Open Systems=18, Open Power Systems=0
15-
[2024-10-14 13:55:53,434 submission_checker1.py:2972 INFO] Network Systems=0, Network Power Systems=0
16-
[2024-10-14 13:55:53,434 submission_checker1.py:2977 INFO] ---
17-
[2024-10-14 13:55:53,434 submission_checker1.py:2982 INFO] SUMMARY: submission looks OK
11+
[2024-10-14 13:56:43,464 submission_checker1.py:2959 INFO] ---
12+
[2024-10-14 13:56:43,464 submission_checker1.py:2961 INFO] Systems=18, Power Systems=0
13+
[2024-10-14 13:56:43,464 submission_checker1.py:2962 INFO] Closed Systems=0, Closed Power Systems=0
14+
[2024-10-14 13:56:43,464 submission_checker1.py:2967 INFO] Open Systems=18, Open Power Systems=0
15+
[2024-10-14 13:56:43,464 submission_checker1.py:2972 INFO] Network Systems=0, Network Power Systems=0
16+
[2024-10-14 13:56:43,464 submission_checker1.py:2977 INFO] ---
17+
[2024-10-14 13:56:43,464 submission_checker1.py:2982 INFO] SUMMARY: submission looks OK
1818
INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4mlops/script/run-mlperf-inference-submission-checker/customize.py
1919
2020
```
@@ -27,7 +27,7 @@ INFO:root: ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4ml
2727
| 3 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config | retinanet | retinanet | Offline | 0.431244 | mAP: 76.951 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config/retinanet/offline | onnxruntime v1.19.2 | Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35) | Automated by MLCommons CM v3.2.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
2828
| 4 | MLCommons | available | open | edge | gh_action | gh_action-reference-gpu-pytorch_v2.4.1-default_config | gptj-99 | gptj-99 | Offline | 50.3446 | ROUGE1: 32.2581 ROUGE2: 6.6667 ROUGEL: 22.5806 GEN_LEN: 264 | 1 | Intel(R) Xeon(R) w7-2495X | 1 | 24 | NVIDIA GeForce RTX 4090 | 1 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.4.1-default_config/gptj-99/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.4. | 1 | 0 | v4.1 | 0 | False | Tokens/s | fp32 |
2929
| 5 | MLCommons | available | open | edge | gh_windows | gh_windows-reference-cpu-tf_v2.17.0-default_config | resnet50 | resnet | Offline | 21.8155 | acc: 76.000 | 1 | undefined | 1 | 1 | nan | 0 | open/MLCommons/results/gh_windows-reference-cpu-tf_v2.17.0-default_config/resnet50/offline | tf v2.17.0 | Windows-2022Server-10.0.20348-SP0 | Automated by MLCommons CM v2.4.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
30-
| 6 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config | bert-99 | bert-99 | Offline | 6.37544 | F1: 80.0 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline | deepsparse v1.8.0 | Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35) | Automated by MLCommons CM v3.2.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
30+
| 6 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config | bert-99 | bert-99 | Offline | 6.17428 | F1: 80.0 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline | deepsparse v1.8.0 | Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35) | Automated by MLCommons CM v3.2.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
3131
| 7 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-reference-cpu-tf_v2.17.0-default_config | resnet50 | resnet | Offline | 21.0119 | acc: 76.000 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-tf_v2.17.0-default_config/resnet50/offline | tf v2.17.0 | Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35) | Automated by MLCommons CM v3.2.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
3232
| 8 | MLCommons | available | open | edge | gh_windows | gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config | resnet50 | resnet | Offline | 17.8036 | acc: 76.000 | 1 | undefined | 1 | 1 | nan | 0 | open/MLCommons/results/gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config/resnet50/offline | onnxruntime v1.19.2 | Windows-2022Server-10.0.20348-SP0 | Automated by MLCommons CM v2.4.0. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |
3333
| 9 | MLCommons | available | open | edge | gh_ubuntu-latest_x86 | gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.4.1-default_config | retinanet | retinanet | Offline | 0.412771 | mAP: 76.951 | 1 | AMD EPYC 7763 64-Core Processor | 1 | 2 | nan | 0 | open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.4.1-default_config/retinanet/offline | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35) | Automated by MLCommons CM v3.2.1. | 1 | 0 | v4.1 | 0 | False | Samples/s | fp32 |

‎summary.csv

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-refe
55
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config,retinanet,retinanet,Offline,0.431244,mAP: 76.951,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.19.2-default_config/retinanet/offline,"onnxruntime v1.19.2","Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35)","Automated by MLCommons CM v3.2.1. ",1,0,v4.1,0,False,Samples/s,"fp32"
66
MLCommons,available,open,"edge","gh_action",gh_action-reference-gpu-pytorch_v2.4.1-default_config,gptj-99,gptj-99,Offline,50.3446,ROUGE1: 32.2581 ROUGE2: 6.6667 ROUGEL: 22.5806 GEN_LEN: 264,1,"Intel(R) Xeon(R) w7-2495X",1,24,"NVIDIA GeForce RTX 4090","1",open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.4.1-default_config/gptj-99/offline,"pytorch v2.4.1","Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35)","Automated by MLCommons CM v2.3.4. ",1,0,v4.1,0,False,Tokens/s,"fp32"
77
MLCommons,available,open,"edge","gh_windows",gh_windows-reference-cpu-tf_v2.17.0-default_config,resnet50,resnet,Offline,21.8155,acc: 76.000,1,"undefined",1,1,"N/A","0",open/MLCommons/results/gh_windows-reference-cpu-tf_v2.17.0-default_config/resnet50/offline,"tf v2.17.0","Windows-2022Server-10.0.20348-SP0","Automated by MLCommons CM v2.4.0. ",1,0,v4.1,0,False,Samples/s,"fp32"
8-
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config,bert-99,bert-99,Offline,6.37544,F1: 80.0,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline,"deepsparse v1.8.0","Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35)","Automated by MLCommons CM v3.2.1. ",1,0,v4.1,0,False,Samples/s,"fp32"
8+
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config,bert-99,bert-99,Offline,6.17428,F1: 80.0,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline,"deepsparse v1.8.0","Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35)","Automated by MLCommons CM v3.2.1. ",1,0,v4.1,0,False,Samples/s,"fp32"
99
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-reference-cpu-tf_v2.17.0-default_config,resnet50,resnet,Offline,21.0119,acc: 76.000,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-tf_v2.17.0-default_config/resnet50/offline,"tf v2.17.0","Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35)","Automated by MLCommons CM v3.2.1. ",1,0,v4.1,0,False,Samples/s,"fp32"
1010
MLCommons,available,open,"edge","gh_windows",gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config,resnet50,resnet,Offline,17.8036,acc: 76.000,1,"undefined",1,1,"N/A","0",open/MLCommons/results/gh_windows-reference-cpu-onnxruntime_v1.19.2-default_config/resnet50/offline,"onnxruntime v1.19.2","Windows-2022Server-10.0.20348-SP0","Automated by MLCommons CM v2.4.0. ",1,0,v4.1,0,False,Samples/s,"fp32"
1111
MLCommons,available,open,"edge","gh_ubuntu-latest_x86",gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.4.1-default_config,retinanet,retinanet,Offline,0.412771,mAP: 76.951,1,"AMD EPYC 7763 64-Core Processor",1,2,"N/A","0",open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.4.1-default_config/retinanet/offline,"pytorch v2.4.1","Ubuntu 22.04 (linux-6.5.0-1025-azure-glibc2.35)","Automated by MLCommons CM v3.2.1. ",1,0,v4.1,0,False,Samples/s,"fp32"

‎summary.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@
189189
"Model":"bert-99",
190190
"MlperfModel":"bert-99",
191191
"Scenario":"Offline",
192-
"Result":6.37544,
192+
"Result":6.17428,
193193
"Accuracy":"F1: 80.0",
194194
"number_of_nodes":1,
195195
"host_processor_model_name":"AMD EPYC 7763 64-Core Processor",

‎summary.xlsx

0 Bytes
Binary file not shown.

‎summary_results.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -428,7 +428,7 @@
428428
"Code": "https://github.com/GATEOverflow/mlperf_inference_test_submissions_v5.0/tree/main/open/MLCommons/code",
429429
"Unique ID (e.g. for Audit)": "edge/open/MLCommons/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99",
430430
"ColorKey": "availableMLCommons",
431-
"Performance_Result": 6.37544,
431+
"Performance_Result": 6.17428,
432432
"Performance_Units": "Samples/s"
433433
},
434434
{

0 commit comments

Comments
 (0)