Sync Dev

.github/workflows/test-mlperf-inference-mixtral.yml

@@ -1,11 +1,8 @@
-# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
-# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
-
 name: MLPerf inference MIXTRAL-8x7B
 
 on:
   schedule:
-    - cron: "59 19 * * *"   # 30th minute and 20th hour => 20:30 UTC => 2 AM IST 
+    - cron: "59 23 * * */5"   # 30th minute and 20th hour => 20:30 UTC => 2 AM IST 
 
 jobs:
   build_reference:

.github/workflows/test-nvidia-mlperf-inference-implementations.yml

@@ -43,6 +43,11 @@ jobs:
             gpu_name=rtx_4090
             docker_string=" --docker"
           fi
+          if [ "${{ matrix.model }}" = "bert-99" ] || [ "${{ matrix.model }}" = "bert-99.9" ]; then
+            category="edge"
+          else
+            category="datacenter,edge"
+          fi
 
           if [ -f "gh_action/bin/deactivate" ]; then source gh_action/bin/deactivate; fi
           python3 -m venv gh_action
@@ -51,6 +56,6 @@ jobs:
           pip install --upgrade mlcflow
           mlc pull repo mlcommons@mlperf-automations --branch=dev
 
-          mlcr --tags=run-mlperf,inference,_all-scenarios,_submission,_full,_r5.0-dev --preprocess_submission=yes --pull_changes=yes --pull_inference_changes=yes --execution_mode=valid --gpu_name=$gpu_name --pull_changes=yes --pull_inference_changes=yes --model=${{ matrix.model }} --submitter="MLCommons" --hw_name=$hw_name --implementation=nvidia --backend=tensorrt --category=datacenter,edge --division=closed  --docker_dt --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --adr.compiler.tags=gcc --device=cuda --use_model_from_host=yes --use_dataset_from_host=yes --results_dir=$HOME/gh_action_results --submission_dir=$HOME/gh_action_submissions --clean  $docker_string --quiet
+          mlcr --tags=run-mlperf,inference,_all-scenarios,_submission,_full,_r5.0-dev --preprocess_submission=yes --pull_changes=yes --pull_inference_changes=yes --execution_mode=valid --gpu_name=$gpu_name --pull_changes=yes --pull_inference_changes=yes --model=${{ matrix.model }} --submitter="MLCommons" --hw_name=$hw_name --implementation=nvidia --backend=tensorrt --category=$category --division=closed  --docker_dt --docker_mlc_repo=mlcommons@mlperf-automations --docker_mlc_repo_branch=dev --adr.compiler.tags=gcc --device=cuda --use_model_from_host=yes --use_dataset_from_host=yes --results_dir=$HOME/gh_action_results --submission_dir=$HOME/gh_action_submissions --clean  $docker_string --quiet
 
           mlcr --tags=push,github,mlperf,inference,submission --repo_url=https://github.com/mlcommons/mlperf_inference_unofficial_submissions_v5.0 --repo_branch=auto-update --commit_message="Results from GH action on NVIDIA_$hw_name" --quiet --submission_dir=$HOME/gh_action_submissions --hw_name=$hw_name

docs/getting-started.md

@@ -1,30 +1,71 @@
+# Getting Started with MLC Script Automation
 
-# Getting Started with CM Script Automation
+## Running MLC Scripts
 
-## Running CM Scripts
-
-To execute a simple script in CM that captures OS details, use the following command:
+To execute a simple script in MLC that captures OS details, use the following command:
 
 ```bash
-cm run script --tags=detect,os -j
+mlcr detect,os -j
 ```
+* Here, `mlcr` is a shortform for `mlc run script --tags=`
 
 This command gathers details about the system on which it's run, such as:
 
 ```json
-{
-    "CM_HOST_OS_TYPE": "linux",
-    "CM_HOST_OS_BITS": "64",
-    "CM_HOST_OS_FLAVOR": "ubuntu",
-    "CM_HOST_OS_FLAVOR_LIKE": "debian",
-    "CM_HOST_OS_VERSION": "24.04",
-    "CM_HOST_OS_KERNEL_VERSION": "6.8.0-45-generic",
-    "CM_HOST_OS_GLIBC_VERSION": "2.39",
-    "CM_HOST_OS_MACHINE": "x86_64",
-    "CM_HOST_OS_PACKAGE_MANAGER": "apt",
-    "CM_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
-    "CM_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
-    "+CM_HOST_OS_DEFAULT_LIBRARY_PATH": [
+$ mlcr detect,os -j
+[2025-02-03 04:57:23,449 main.py:694 INFO] - Repos path for Index: /home/arjun/MLC/repos
+[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for script saved to /home/arjun/MLC/repos/index_script.json.
+[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for cache saved to /home/arjun/MLC/repos/index_cache.json.
+[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for experiment saved to /home/arjun/MLC/repos/index_experiment.json.
+[2025-02-03 04:57:24,210 module.py:574 INFO] - * mlcr detect,os
+[2025-02-03 04:57:24,213 module.py:5354 INFO] -        ! cd /mnt/arjun/MLC/repos/gateoverflow@mlperf-automations
+[2025-02-03 04:57:24,213 module.py:5355 INFO] -        ! call /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
+[2025-02-03 04:57:24,245 module.py:5501 INFO] -        ! call "postprocess" from /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os/customize.py
+[2025-02-03 04:57:24,254 module.py:2195 INFO] - {
+  "return": 0,
+  "env": {
+    "MLC_HOST_OS_TYPE": "linux",
+    "MLC_HOST_OS_BITS": "64",
+    "MLC_HOST_OS_FLAVOR": "ubuntu",
+    "MLC_HOST_OS_FLAVOR_LIKE": "debian",
+    "MLC_HOST_OS_VERSION": "24.04",
+    "MLC_HOST_OS_KERNEL_VERSION": "6.8.0-52-generic",
+    "MLC_HOST_OS_GLIBC_VERSION": "2.39",
+    "MLC_HOST_OS_MACHINE": "x86_64",
+    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
+    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
+    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
+    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
+      "/usr/local/lib/x86_64-linux-gnu",
+      "/lib/x86_64-linux-gnu",
+      "/usr/lib/x86_64-linux-gnu",
+      "/usr/lib/x86_64-linux-gnu64",
+      "/usr/local/lib64",
+      "/lib64",
+      "/usr/lib64",
+      "/usr/local/lib",
+      "/lib",
+      "/usr/lib",
+      "/usr/x86_64-linux-gnu/lib64",
+      "/usr/x86_64-linux-gnu/lib"
+    ],
+    "MLC_HOST_PLATFORM_FLAVOR": "x86_64",
+    "MLC_HOST_PYTHON_BITS": "64",
+    "MLC_HOST_SYSTEM_NAME": "arjun-spr"
+  },
+  "new_env": {
+    "MLC_HOST_OS_TYPE": "linux",
+    "MLC_HOST_OS_BITS": "64",
+    "MLC_HOST_OS_FLAVOR": "ubuntu",
+    "MLC_HOST_OS_FLAVOR_LIKE": "debian",
+    "MLC_HOST_OS_VERSION": "24.04",
+    "MLC_HOST_OS_KERNEL_VERSION": "6.8.0-52-generic",
+    "MLC_HOST_OS_GLIBC_VERSION": "2.39",
+    "MLC_HOST_OS_MACHINE": "x86_64",
+    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
+    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
+    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
+    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
       "/usr/local/lib/x86_64-linux-gnu",
       "/lib/x86_64-linux-gnu",
       "/usr/lib/x86_64-linux-gnu",
@@ -38,98 +79,24 @@ This command gathers details about the system on which it's run, such as:
       "/usr/x86_64-linux-gnu/lib64",
       "/usr/x86_64-linux-gnu/lib"
     ],
-    "CM_HOST_PLATFORM_FLAVOR": "x86_64",
-    "CM_HOST_PYTHON_BITS": "64",
-    "CM_HOST_SYSTEM_NAME": "intel-spr-i9"
+    "MLC_HOST_PLATFORM_FLAVOR": "x86_64",
+    "MLC_HOST_PYTHON_BITS": "64",
+    "MLC_HOST_SYSTEM_NAME": "arjun-spr"
+  },
+  "state": {
+    "os_uname_machine": "x86_64",
+    "os_uname_all": "Linux arjun-spr 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux"
+  },
+  "new_state": {
+    "os_uname_machine": "x86_64",
+    "os_uname_all": "Linux arjun-spr 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux"
+  },
+  "deps": []
 }
 ```
 
-For more details on CM scripts, see the [CM documentation](index.md).
-
-### Adding New CM Scripts
+For more details on MLC scripts, see the [MLC documentation](index.md).
 
-CM aims to provide lightweight connectors between existing automation scripts and tools without substituting them. You can add your own scripts to CM with the following command, which creates a script named `hello-world`:
-
-```bash
-cm add script hello-world --tags=hello-world,display,test
-```
-
-This command initializes a CM script in the local repository with the following structure:
-
-```
-└── CM
-    ├── index.json
-    ├── repos
-    │   ├── local
-    │   │   ├── cfg
-    │   │   ├── cache
-    │   │   ├── cmr.yaml
-    │   │   └── script
-    │   │       └── hello-world
-    │   │           ├── _cm.yaml
-    │   │           ├── customize.py
-    │   │           ├── README-extra.md
-    │   │           ├── run.bat
-    │   │           └── run.sh
-    │   └── mlcommons@cm4mlops
-    └── repos.json
-```
 
 You can also execute the script from Python as follows:
 
-```python
-import cmind
-output = cmind.access({'action':'run', 'automation':'script', 'tags':'hello-world,display,test'})
-if output['return'] == 0:
-    print(output)
-```
-
-If you discover that your new script is similar to an existing script in any CM repository, you can clone an existing script using the following command:
-
-```bash
-cm copy script <source_script> .:<target_script>
-```
-
-Here, `<source_script>` is the name of the existing script, and `<target_script>` is the name of the new script you're creating. Existing script names in the `cm4mlops` repository can be found [here](https://github.com/mlcommons/cm4mlops/tree/mlperf-inference/script).
-
-## Caching and Reusing CM Script Outputs
-
-By default, CM scripts run in the current directory and record all new files there. For example, a universal download script might download an image to the current directory:
-
-```bash
-cm run script --tags=download,file,_wget --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e
-```
-
-To cache and reuse the output of scripts, CM offers a `cache` automation feature similar to `script`. When `"cache":true` is specified in a script's metadata, CM will create a `cache` directory in `$HOME/CM/repos/local` with a unique ID and the same tags as `script`, and execute the script there.
-
-Subsequent executions of the same script will reuse files from the cache, avoiding redundancy. This is especially useful for large files or data sets.
-
-You can manage cache entries and find specific ones using commands like:
-
-```bash
-cm show cache
-cm show cache --tags=get,ml-model,resnet50,_onnx
-cm find cache --tags=download,file,ml-model,resnet50,_onnx
-cm info cache --tags=download,file,ml-model,resnet50,_onnx
-```
-
-To clean cache entries:
-
-```bash
-cm rm cache --tags=ml-model,resnet50
-cm rm cache -f  # Clean all entries
-```
-
-You can completely reset the CM framework by removing the `$HOME/CM` directory, which deletes all downloaded repositories and cached entries.
-
-## Integration with Containers
-
-CM scripts are designed to run natively or inside containers with the same commands. You can substitute `cm run script` with `cm docker script` to execute a script inside an automatically-generated container:
-
-```bash
-cm docker script --tags=python,app,image-classification,onnx,_cpu
-```
-
-CM automatically handles the generation of Dockerfiles, building of containers, and execution within containers, providing a seamless experience whether running scripts natively or in containers. 
-
-This approach simplifies the development process by eliminating the need for separate Dockerfile maintenance and allows for the use of native scripts and workflows directly within containers.