Skip to content

Commit 76e275c

Browse files
anandhu-enganandhu-eng
authored
Merge pull request #154 from mlcommons/dev
Sync Dev
2 parents 97c37c4 + f84d237 commit 76e275c

File tree

32 files changed

+295
-124
lines changed

32 files changed

+295
-124
lines changed

‎.github/workflows/build_wheel.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ on:
99
- dev
1010
paths:
1111
- VERSION
12-
- setup.py
1312

1413
jobs:
1514
build_wheels:

‎.github/workflows/check-broken-links.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,12 @@ on:
88
jobs:
99
markdown-link-check:
1010
runs-on: ubuntu-latest
11+
1112
# check out the latest version of the code
1213
steps:
1314
- uses: actions/checkout@v4
1415

16+
1517
# Checks the status of hyperlinks in .md files in verbose mode
1618
- name: Check links
1719
uses: gaurav-nelson/github-action-markdown-link-check@v1

‎.github/workflows/test-mlc-script-features.yml

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -35,42 +35,44 @@ jobs:
3535
3636
- name: Test Python venv
3737
run: |
38-
mlc run script --tags=install,python-venv --name=test --quiet
38+
mlcr --tags=install,python-venv --name=test --quiet
3939
mlc search cache --tags=get,python,virtual,name-test --quiet
4040
4141
- name: Test variations
4242
run: |
43-
mlc run script --tags=get,dataset,preprocessed,imagenet,_NHWC --quiet
43+
mlcr --tags=get,dataset,preprocessed,imagenet,_NHWC --quiet
4444
mlc search cache --tags=get,dataset,preprocessed,imagenet,-_NCHW
4545
mlc search cache --tags=get,dataset,preprocessed,imagenet,-_NHWC
4646
4747
- name: Test versions
4848
continue-on-error: true
4949
if: runner.os == 'linux'
5050
run: |
51-
mlc run script --tags=get,generic-python-lib,_package.scipy --version=1.9.3 --quiet
51+
mlcr --tags=get,generic-python-lib,_package.scipy --version=1.9.3 --quiet
5252
test $? -eq 0 || exit $?
53-
mlc run script --tags=get,generic-python-lib,_package.scipy --version=1.9.2 --quiet
53+
mlcr --tags=get,generic-python-lib,_package.scipy --version=1.9.2 --quiet
5454
test $? -eq 0 || exit $?
55-
mlc run script --tags=get,generic-python-lib,_package.scipy --version=1.9.3 --quiet --only_execute_from_cache=True
56-
test $? -eq 0 || exit 0
55+
# Need to add find cache here
56+
# mlcr --tags=get,generic-python-lib,_package.scipy --version=1.9.3 --quiet --only_execute_from_cache=True
57+
# test $? -eq 0 || exit 0
5758
5859
- name: Test python install from src
5960
run: |
60-
mlc run script --tags=python,src,install,_shared --version=3.9.10 --quiet
61+
mlcr --tags=python,src,install,_shared --version=3.9.10 --quiet
6162
mlc search cache --tags=python,src,install,_shared,version-3.9.10
6263
6364
- name: Run docker container from dockerhub on linux
6465
if: runner.os == 'linux'
6566
run: |
66-
mlc run script --tags=run,docker,container --adr.compiler.tags=gcc --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --image_name=cm-script-app-image-classification-onnx-py --env.MLC_DOCKER_RUN_SCRIPT_TAGS=app,image-classification,onnx,python --env.MLC_DOCKER_IMAGE_BASE=ubuntu:22.04 --env.MLC_DOCKER_IMAGE_REPO=cknowledge --quiet
67+
mlcr --tags=run,docker,container --adr.compiler.tags=gcc --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --image_name=cm-script-app-image-classification-onnx-py --env.MLC_DOCKER_RUN_SCRIPT_TAGS=app,image-classification,onnx,python --env.MLC_DOCKER_IMAGE_BASE=ubuntu:22.04 --env.MLC_DOCKER_IMAGE_REPO=cknowledge --quiet
6768
6869
- name: Run docker container locally on linux
6970
if: runner.os == 'linux'
7071
run: |
71-
mlc run script --tags=run,docker,container --adr.compiler.tags=gcc --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --image_name=mlc-script-app-image-classification-onnx-py --env.MLC_DOCKER_RUN_SCRIPT_TAGS=app,image-classification,onnx,python --env.MLC_DOCKER_IMAGE_BASE=ubuntu:22.04 --env.MLC_DOCKER_IMAGE_REPO=local --quiet
72+
mlcr --tags=run,docker,container --adr.compiler.tags=gcc --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --image_name=mlc-script-app-image-classification-onnx-py --env.MLC_DOCKER_RUN_SCRIPT_TAGS=app,image-classification,onnx,python --env.MLC_DOCKER_IMAGE_BASE=ubuntu:22.04 --env.MLC_DOCKER_IMAGE_REPO=local --quiet
7273
7374
- name: Run MLPerf Inference Retinanet with native and virtual Python
75+
if: runner.os == 'linux'
7476
run: |
7577
mlcr --tags=app,mlperf,inference,generic,_cpp,_retinanet,_onnxruntime,_cpu --adr.python.version_min=3.8 --adr.compiler.tags=gcc --adr.openimages-preprocessed.tags=_50 --scenario=Offline --mode=accuracy --test_query_count=10 --rerun --quiet
7678

‎.github/workflows/test-mlperf-inference-abtf-poc.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ jobs:
1818
python-version: [ "3.8", "3.12" ]
1919
backend: [ "pytorch" ]
2020
implementation: [ "python" ]
21-
docker: [ "", " --docker --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --docker_dt" ]
21+
docker: [ "", " --docker --docker_mlc_repo=${{ github.event.pull_request.head.repo.html_url }} --docker_mlc_repo_branch=${{ github.event.pull_request.head.ref }} --docker_dt" ]
2222
extra-args: [ "--adr.compiler.tags=gcc", "--env.MLC_MLPERF_LOADGEN_BUILD_FROM_SRC=off" ]
2323
exclude:
2424
- os: ubuntu-24.04
@@ -28,16 +28,16 @@ jobs:
2828
- os: windows-latest
2929
extra-args: "--adr.compiler.tags=gcc"
3030
- os: windows-latest
31-
docker: " --docker --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --docker_dt"
31+
docker: " --docker --docker_mlc_repo=${{ github.event.pull_request.head.repo.html_url }} --docker_mlc_repo_branch=${{ github.event.pull_request.head.ref }} --docker_dt"
3232
# windows docker image is not supported in CM yet
3333
- os: macos-latest
3434
python-version: "3.8"
3535
- os: macos-13
3636
python-version: "3.8"
3737
- os: macos-latest
38-
docker: " --docker --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --docker_dt"
38+
docker: " --docker --docker_mlc_repo=${{ github.event.pull_request.head.repo.html_url }} --docker_mlc_repo_branch=${{ github.event.pull_request.head.ref }} --docker_dt"
3939
- os: macos-13
40-
docker: " --docker --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --docker_dt"
40+
docker: " --docker --docker_mlc_repo=${{ github.event.pull_request.head.repo.html_url }} --docker_mlc_repo_branch=${{ github.event.pull_request.head.ref }} --docker_dt"
4141

4242
steps:
4343
- uses: actions/checkout@v3

‎.github/workflows/test-mlperf-inference-bert-deepsparse-tf-onnxruntime-pytorch.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,18 @@ jobs:
4747
if: matrix.os != 'windows-latest'
4848
run: |
4949
mlcr --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --submitter="MLCommons" --hw_name=gh_${{ matrix.os }}_x86 --model=bert-99 --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=5 --precision=${{ matrix.precision }} --target_qps=1 -v --quiet
50+
- name: Randomly Execute Step
51+
id: random-check
52+
run: |
53+
RANDOM_NUMBER=$((RANDOM % 10))
54+
echo "Random number is $RANDOM_NUMBER"
55+
if [ "$RANDOM_NUMBER" -eq 0 ]; then
56+
echo "run_step=true" >> $GITHUB_ENV
57+
else
58+
echo "run_step=false" >> $GITHUB_ENV
59+
fi
5060
- name: Retrieve secrets from Keeper
61+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
5162
id: ksecrets
5263
uses: Keeper-Security/ksm-action@master
5364
with:
@@ -57,6 +68,7 @@ jobs:
5768
- name: Push Results
5869
env:
5970
GITHUB_TOKEN: ${{ env.PAT }}
71+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
6072
run: |
6173
git config --global user.name "mlcommons-bot"
6274
git config --global user.email "mlcommons-bot@users.noreply.github.com"

‎.github/workflows/test-mlperf-inference-mlcommons-cpp-resnet50.yml

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -48,12 +48,30 @@ jobs:
4848
if: matrix.os != 'windows-latest'
4949
run: |
5050
mlcr --tags=app,mlperf,inference,mlcommons,cpp --submitter="MLCommons" --hw_name=gh_${{ matrix.os }} -v --quiet
51+
- name: Randomly Execute Step
52+
id: random-check
53+
run: |
54+
RANDOM_NUMBER=$((RANDOM % 10))
55+
echo "Random number is $RANDOM_NUMBER"
56+
if [ "$RANDOM_NUMBER" -eq 0 ]; then
57+
echo "run_step=true" >> $GITHUB_ENV
58+
else
59+
echo "run_step=false" >> $GITHUB_ENV
60+
fi
61+
- name: Retrieve secrets from Keeper
62+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
63+
id: ksecrets
64+
uses: Keeper-Security/ksm-action@master
65+
with:
66+
keeper-secret-config: ${{ secrets.KSM_CONFIG }}
67+
secrets: |-
68+
ubwkjh-Ii8UJDpG2EoU6GQ/field/Access Token > env:PAT
5169
- name: Push Results
52-
if: github.repository_owner == 'gateoverflow'
5370
env:
54-
USER: "GitHub Action"
55-
EMAIL: "admin@gateoverflow.com"
56-
GITHUB_TOKEN: ${{ secrets.TEST_RESULTS_GITHUB_TOKEN }}
71+
GITHUB_TOKEN: ${{ env.PAT }}
72+
USER: mlcommons-bot
73+
EMAIL: mlcommons-bot@users.noreply.github.com
74+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
5775
run: |
5876
git config --global user.name "${{ env.USER }}"
5977
git config --global user.email "${{ env.EMAIL }}"

‎.github/workflows/test-mlperf-inference-resnet50.yml

Lines changed: 13 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,18 @@ jobs:
5858
if: matrix.os != 'windows-latest'
5959
run: |
6060
mlcr --tags=run-mlperf,inference,_submission,_short --submitter="MLCommons" --pull_changes=yes --pull_inference_changes=yes --hw_name=gh_${{ matrix.os }}_x86 --model=resnet50 --implementation=${{ matrix.implementation }} --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --target_qps=1 -v --quiet
61+
- name: Randomly Execute Step
62+
id: random-check
63+
run: |
64+
RANDOM_NUMBER=$((RANDOM % 10))
65+
echo "Random number is $RANDOM_NUMBER"
66+
if [ "$RANDOM_NUMBER" -eq 0 ]; then
67+
echo "run_step=true" >> $GITHUB_ENV
68+
else
69+
echo "run_step=false" >> $GITHUB_ENV
70+
fi
6171
- name: Retrieve secrets from Keeper
62-
if: github.repository_owner == 'mlcommons'
72+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
6373
id: ksecrets
6474
uses: Keeper-Security/ksm-action@master
6575
with:
@@ -69,7 +79,7 @@ jobs:
6979
- name: Push Results
7080
env:
7181
GITHUB_TOKEN: ${{ env.PAT }}
72-
if: github.repository_owner == 'mlcommons'
82+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
7383
run: |
7484
git config --global user.name "mlcommons-bot"
7585
git config --global user.email "mlcommons-bot@users.noreply.github.com"
@@ -78,15 +88,4 @@ jobs:
7888
git config --global credential.https://gist.github.com.helper ""
7989
git config --global credential.https://gist.github.com.helper "!gh auth git-credential"
8090
mlcr --tags=push,github,mlperf,inference,submission --repo_url=https://github.com/mlcommons/mlperf_inference_test_submissions_v5.0 --repo_branch=auto-update --commit_message="Results from R50 GH action on ${{ matrix.os }}" --quiet
81-
- name: Push Results
82-
env:
83-
GITHUB_TOKEN: ${{ secrets.PAT1 }}
84-
if: github.repository_owner == 'gateoverflow'
85-
run: |
86-
git config --global user.name "mlcommons-bot"
87-
git config --global user.email "mlcommons-bot@users.noreply.github.com"
88-
git config --global credential.https://github.com.helper ""
89-
git config --global credential.https://github.com.helper "!gh auth git-credential"
90-
git config --global credential.https://gist.github.com.helper ""
91-
git config --global credential.https://gist.github.com.helper "!gh auth git-credential"
92-
mlcr --tags=push,github,mlperf,inference,submission --repo_url=https://github.com/gateoverflow/mlperf_inference_test_submissions_v5.0 --repo_branch=auto-update --commit_message="Results from R50 GH action on ${{ matrix.os }}" --quiet
91+

‎.github/workflows/test-mlperf-inference-retinanet.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,18 @@ jobs:
5252
if: matrix.os != 'windows-latest'
5353
run: |
5454
mlcr --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --submitter="MLCommons" --pull_changes=yes --pull_inference_changes=yes --hw_name=gh_${{ matrix.os }}_x86 --model=retinanet --implementation=${{ matrix.implementation }} --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=5 --quiet -v --target_qps=1
55+
- name: Randomly Execute Step
56+
id: random-check
57+
run: |
58+
RANDOM_NUMBER=$((RANDOM % 10))
59+
echo "Random number is $RANDOM_NUMBER"
60+
if [ "$RANDOM_NUMBER" -eq 0 ]; then
61+
echo "run_step=true" >> $GITHUB_ENV
62+
else
63+
echo "run_step=false" >> $GITHUB_ENV
64+
fi
5565
- name: Retrieve secrets from Keeper
66+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
5667
id: ksecrets
5768
uses: Keeper-Security/ksm-action@master
5869
with:
@@ -62,6 +73,7 @@ jobs:
6273
- name: Push Results
6374
env:
6475
GITHUB_TOKEN: ${{ env.PAT }}
76+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
6577
run: |
6678
git config --global user.name "mlcommons-bot"
6779
git config --global user.email "mlcommons-bot@users.noreply.github.com"

‎.github/workflows/test-mlperf-inference-tvm-resnet50.yml

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,19 @@ jobs:
3434
mlcr --quiet --tags=get,sys-utils-cm
3535
- name: Test MLC Tutorial TVM
3636
run: |
37-
mlcr --tags=run-mlperf,inference,_submission,_short --adr.python.name=mlperf --adr.python.version_min=3.8 --submitter=Community --implementation=python --hw_name=default --model=resnet50 --backend=tvm-onnx --device=cpu --scenario=Offline --mode=accuracy --test_query_count=5 --clean --quiet ${{ matrix.extra-options }}
37+
mlcr --tags=run-mlperf,inference,_submission,_short --adr.python.name=mlperf --adr.python.version_min=3.8 --submitter=MLCommons --implementation=python --hw_name=gh_ubuntu-latest --model=resnet50 --backend=tvm-onnx --device=cpu --scenario=Offline --mode=accuracy --test_query_count=5 --clean --quiet ${{ matrix.extra-options }}
38+
- name: Randomly Execute Step
39+
id: random-check
40+
run: |
41+
RANDOM_NUMBER=$((RANDOM % 10))
42+
echo "Random number is $RANDOM_NUMBER"
43+
if [ "$RANDOM_NUMBER" -eq 0 ]; then
44+
echo "run_step=true" >> $GITHUB_ENV
45+
else
46+
echo "run_step=false" >> $GITHUB_ENV
47+
fi
3848
- name: Retrieve secrets from Keeper
49+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
3950
id: ksecrets
4051
uses: Keeper-Security/ksm-action@master
4152
with:
@@ -45,9 +56,12 @@ jobs:
4556
- name: Push Results
4657
env:
4758
GITHUB_TOKEN: ${{ env.PAT }}
59+
USER: mlcommons-bot
60+
EMAIL: mlcommons-bot@users.noreply.github.com
61+
if: github.repository_owner == 'mlcommons' && env.run_step == 'true'
4862
run: |
49-
git config --global user.name "mlcommons-bot"
50-
git config --global user.email "mlcommons-bot@users.noreply.github.com"
63+
git config --global user.name "${{ env.USER }}"
64+
git config --global user.email "${{ env.EMAIL }}"
5165
git config --global credential.https://github.com.helper ""
5266
git config --global credential.https://github.com.helper "!gh auth git-credential"
5367
git config --global credential.https://gist.github.com.helper ""

‎README.md

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
[![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md)
44
[![Downloads](https://static.pepy.tech/badge/mlcflow)](https://pepy.tech/project/mlcflow)
5-
[![MLC Script Automation Test](https://github.com/mlcommons/mlperf-automations/actions/workflows/test-mlc-script-features.yml/badge.svg)](https://github.com/mlcommons/mlperf-automations/actions/workflows/test-mlc-script-features.yml)
5+
[![MLC script automation features test](https://github.com/mlcommons/mlperf-automations/actions/workflows/test-mlc-script-features.yml/badge.svg?cache-bust=1)](https://github.com/mlcommons/mlperf-automations/actions/workflows/test-mlc-script-features.yml)
66
[![MLPerf Inference ABTF POC Test](https://github.com/mlcommons/mlperf-automations/actions/workflows/test-mlperf-inference-abtf-poc.yml/badge.svg)](https://github.com/mlcommons/mlperf-automations/actions/workflows/test-mlperf-inference-abtf-poc.yml)
77

88

@@ -23,12 +23,9 @@ Starting **January 2025**, MLPerf automation scripts are built on the powerful [
2323

2424
## 🧰 MLCFlow (MLC) Automations
2525

26-
Building on the foundation of its predecessor, the **Collective Mind (CM)** framework, MLCFlow takes ML workflows to the next level by streamlining complex tasks like Docker container management and caching. The `mlcflow` package, written in Python, provides seamless support through both a command-line interface (CLI) and an API, making it easy to access and manage automation scripts.
27-
28-
### Core Automations
29-
- **Script Automation** – Automates script execution across different environments.
30-
- **Cache Management** – Manages reusable cached results to accelerate workflow processes.
26+
Building upon the robust foundation of its predecessor, the Collective Mind (CM) framework, MLCFlow elevates machine learning workflows by simplifying complex tasks such as Docker container management and caching. Written in Python, the mlcflow package offers a versatile interface, supporting both a user-friendly command-line interface (CLI) and a flexible API for effortless automation script management.
3127

28+
At its core, MLCFlow relies on a single powerful automation, the Script, which is extended by two actions: CacheAction and DockerAction. Together, these components provide streamlined functionality to optimize and enhance your ML workflow automation experience.
3229

3330
---
3431

@@ -40,10 +37,17 @@ We welcome contributions from the community! To contribute:
4037

4138
Your contributions help drive the project forward!
4239

40+
41+
---
42+
43+
## 💬 Join the Discussion
44+
Connect with us on the [MLCommons Benchmark Infra Discord channel](https://discord.gg/T9rHVwQFNX) to engage in discussions about **MLCFlow** and **MLPerf Automations**. We’d love to hear your thoughts, questions, and ideas!
45+
4346
---
4447

45-
## 📰 News
46-
Stay tuned for upcoming updates and announcements.
48+
## 📰 Stay Updated
49+
Keep track of the latest development progress and tasks on our [MLPerf Automations Development Board](https://github.com/orgs/mlcommons/projects/50/views/7?sliceBy%5Bvalue%5D=_noValue).
50+
Stay tuned for exciting updates and announcements!
4751

4852
---
4953

0 commit comments

Comments
 (0)