Cleaned old docs

Author: arjunsuresh

docs/getting-started.md

@@ -1,30 +1,71 @@
+# Getting Started with MLC Script Automation
 
-# Getting Started with CM Script Automation
+## Running MLC Scripts
 
-## Running CM Scripts
-
-To execute a simple script in CM that captures OS details, use the following command:
+To execute a simple script in MLC that captures OS details, use the following command:
 
 ```bash
-cm run script --tags=detect,os -j
+mlcr detect,os -j
 ```
+* Here, `mlcr` is a shortform for `mlc run script --tags=`
 
 This command gathers details about the system on which it's run, such as:
 
 ```json
-{
-    "CM_HOST_OS_TYPE": "linux",
-    "CM_HOST_OS_BITS": "64",
-    "CM_HOST_OS_FLAVOR": "ubuntu",
-    "CM_HOST_OS_FLAVOR_LIKE": "debian",
-    "CM_HOST_OS_VERSION": "24.04",
-    "CM_HOST_OS_KERNEL_VERSION": "6.8.0-45-generic",
-    "CM_HOST_OS_GLIBC_VERSION": "2.39",
-    "CM_HOST_OS_MACHINE": "x86_64",
-    "CM_HOST_OS_PACKAGE_MANAGER": "apt",
-    "CM_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
-    "CM_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
-    "+CM_HOST_OS_DEFAULT_LIBRARY_PATH": [
+$ mlcr detect,os -j
+[2025-02-03 04:57:23,449 main.py:694 INFO] - Repos path for Index: /home/arjun/MLC/repos
+[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for script saved to /home/arjun/MLC/repos/index_script.json.
+[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for cache saved to /home/arjun/MLC/repos/index_cache.json.
+[2025-02-03 04:57:24,167 main.py:837 INFO] - Shared index for experiment saved to /home/arjun/MLC/repos/index_experiment.json.
+[2025-02-03 04:57:24,210 module.py:574 INFO] - * mlcr detect,os
+[2025-02-03 04:57:24,213 module.py:5354 INFO] -        ! cd /mnt/arjun/MLC/repos/gateoverflow@mlperf-automations
+[2025-02-03 04:57:24,213 module.py:5355 INFO] -        ! call /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
+[2025-02-03 04:57:24,245 module.py:5501 INFO] -        ! call "postprocess" from /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os/customize.py
+[2025-02-03 04:57:24,254 module.py:2195 INFO] - {
+  "return": 0,
+  "env": {
+    "MLC_HOST_OS_TYPE": "linux",
+    "MLC_HOST_OS_BITS": "64",
+    "MLC_HOST_OS_FLAVOR": "ubuntu",
+    "MLC_HOST_OS_FLAVOR_LIKE": "debian",
+    "MLC_HOST_OS_VERSION": "24.04",
+    "MLC_HOST_OS_KERNEL_VERSION": "6.8.0-52-generic",
+    "MLC_HOST_OS_GLIBC_VERSION": "2.39",
+    "MLC_HOST_OS_MACHINE": "x86_64",
+    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
+    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
+    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
+    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
+      "/usr/local/lib/x86_64-linux-gnu",
+      "/lib/x86_64-linux-gnu",
+      "/usr/lib/x86_64-linux-gnu",
+      "/usr/lib/x86_64-linux-gnu64",
+      "/usr/local/lib64",
+      "/lib64",
+      "/usr/lib64",
+      "/usr/local/lib",
+      "/lib",
+      "/usr/lib",
+      "/usr/x86_64-linux-gnu/lib64",
+      "/usr/x86_64-linux-gnu/lib"
+    ],
+    "MLC_HOST_PLATFORM_FLAVOR": "x86_64",
+    "MLC_HOST_PYTHON_BITS": "64",
+    "MLC_HOST_SYSTEM_NAME": "arjun-spr"
+  },
+  "new_env": {
+    "MLC_HOST_OS_TYPE": "linux",
+    "MLC_HOST_OS_BITS": "64",
+    "MLC_HOST_OS_FLAVOR": "ubuntu",
+    "MLC_HOST_OS_FLAVOR_LIKE": "debian",
+    "MLC_HOST_OS_VERSION": "24.04",
+    "MLC_HOST_OS_KERNEL_VERSION": "6.8.0-52-generic",
+    "MLC_HOST_OS_GLIBC_VERSION": "2.39",
+    "MLC_HOST_OS_MACHINE": "x86_64",
+    "MLC_HOST_OS_PACKAGE_MANAGER": "apt",
+    "MLC_HOST_OS_PACKAGE_MANAGER_INSTALL_CMD": "DEBIAN_FRONTEND=noninteractive apt-get install -y",
+    "MLC_HOST_OS_PACKAGE_MANAGER_UPDATE_CMD": "apt-get update -y",
+    "+MLC_HOST_OS_DEFAULT_LIBRARY_PATH": [
       "/usr/local/lib/x86_64-linux-gnu",
       "/lib/x86_64-linux-gnu",
       "/usr/lib/x86_64-linux-gnu",
@@ -38,98 +79,24 @@ This command gathers details about the system on which it's run, such as:
       "/usr/x86_64-linux-gnu/lib64",
       "/usr/x86_64-linux-gnu/lib"
     ],
-    "CM_HOST_PLATFORM_FLAVOR": "x86_64",
-    "CM_HOST_PYTHON_BITS": "64",
-    "CM_HOST_SYSTEM_NAME": "intel-spr-i9"
+    "MLC_HOST_PLATFORM_FLAVOR": "x86_64",
+    "MLC_HOST_PYTHON_BITS": "64",
+    "MLC_HOST_SYSTEM_NAME": "arjun-spr"
+  },
+  "state": {
+    "os_uname_machine": "x86_64",
+    "os_uname_all": "Linux arjun-spr 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux"
+  },
+  "new_state": {
+    "os_uname_machine": "x86_64",
+    "os_uname_all": "Linux arjun-spr 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux"
+  },
+  "deps": []
 }
 ```
 
-For more details on CM scripts, see the [CM documentation](index.md).
-
-### Adding New CM Scripts
+For more details on MLC scripts, see the [MLC documentation](index.md).
 
-CM aims to provide lightweight connectors between existing automation scripts and tools without substituting them. You can add your own scripts to CM with the following command, which creates a script named `hello-world`:
-
-```bash
-cm add script hello-world --tags=hello-world,display,test
-```
-
-This command initializes a CM script in the local repository with the following structure:
-
-```
-└── CM
-    ├── index.json
-    ├── repos
-    │   ├── local
-    │   │   ├── cfg
-    │   │   ├── cache
-    │   │   ├── cmr.yaml
-    │   │   └── script
-    │   │       └── hello-world
-    │   │           ├── _cm.yaml
-    │   │           ├── customize.py
-    │   │           ├── README-extra.md
-    │   │           ├── run.bat
-    │   │           └── run.sh
-    │   └── mlcommons@cm4mlops
-    └── repos.json
-```
 
 You can also execute the script from Python as follows:
 
-```python
-import cmind
-output = cmind.access({'action':'run', 'automation':'script', 'tags':'hello-world,display,test'})
-if output['return'] == 0:
-    print(output)
-```
-
-If you discover that your new script is similar to an existing script in any CM repository, you can clone an existing script using the following command:
-
-```bash
-cm copy script <source_script> .:<target_script>
-```
-
-Here, `<source_script>` is the name of the existing script, and `<target_script>` is the name of the new script you're creating. Existing script names in the `cm4mlops` repository can be found [here](https://github.com/mlcommons/cm4mlops/tree/mlperf-inference/script).
-
-## Caching and Reusing CM Script Outputs
-
-By default, CM scripts run in the current directory and record all new files there. For example, a universal download script might download an image to the current directory:
-
-```bash
-cm run script --tags=download,file,_wget --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e
-```
-
-To cache and reuse the output of scripts, CM offers a `cache` automation feature similar to `script`. When `"cache":true` is specified in a script's metadata, CM will create a `cache` directory in `$HOME/CM/repos/local` with a unique ID and the same tags as `script`, and execute the script there.
-
-Subsequent executions of the same script will reuse files from the cache, avoiding redundancy. This is especially useful for large files or data sets.
-
-You can manage cache entries and find specific ones using commands like:
-
-```bash
-cm show cache
-cm show cache --tags=get,ml-model,resnet50,_onnx
-cm find cache --tags=download,file,ml-model,resnet50,_onnx
-cm info cache --tags=download,file,ml-model,resnet50,_onnx
-```
-
-To clean cache entries:
-
-```bash
-cm rm cache --tags=ml-model,resnet50
-cm rm cache -f  # Clean all entries
-```
-
-You can completely reset the CM framework by removing the `$HOME/CM` directory, which deletes all downloaded repositories and cached entries.
-
-## Integration with Containers
-
-CM scripts are designed to run natively or inside containers with the same commands. You can substitute `cm run script` with `cm docker script` to execute a script inside an automatically-generated container:
-
-```bash
-cm docker script --tags=python,app,image-classification,onnx,_cpu
-```
-
-CM automatically handles the generation of Dockerfiles, building of containers, and execution within containers, providing a seamless experience whether running scripts natively or in containers. 
-
-This approach simplifies the development process by eliminating the need for separate Dockerfile maintenance and allows for the use of native scripts and workflows directly within containers.

docs/index.md

@@ -1,29 +1,29 @@
-# CM "script" automation specification
+# MLC "script" automation specification
 
-Please check the [CM documentation](https://docs.mlcommons.org/ck) for more details about the CM automation language.
+Please check the [MLC documentation](https://docs.mlcommons.org/mlcflow) for more details about the MLCflow interface.
 
-See the [automatically generated catalog](scripts/index.md) of all CM scripts from MLCommons.
+See the [automatically generated catalog](scripts/index.md) of all the MLC scripts.
 
-## Understanding CM scripts
+## Understanding MLC scripts
 
-* A CM script is identified by a set of tags and by unique ID. 
-* Further each CM script can have multiple variations and they are identified by variation tags which are treated in the same way as tags and identified by a `_` prefix.
+* An MLC script is identified by a set of tags and by an unique ID. 
+* Further each MLC script can have multiple variations and they are identified by variation tags which are treated in the same way as tags and identified by a `_` prefix.
 
-### CM script execution flow
+### MLC script execution flow
 <img src="https://github.com/mlcommons/cm4mlops/raw/mlperf-inference/automation/script/assets/scripts-workflow.png" width="248">
 
-* When a CM script is invoked (either by tags or by unique ID), its `_cm.json` is processed first which will check for any `deps` script and if there are, then they are executed in order.
+* When an MLC script is invoked (either by tags or by unique ID), its `meta.yaml` is processed first which will check for any `deps` script and if there are, then they are executed in order.
 * Once all the `deps` scripts are executed, `customize.py` file is checked and if existing `preprocess` function inside it is executed if present. 
-* Then any `prehook_deps` CM scripts mentioned in `_cm.json` are executed similar to `deps`
+* Then any `prehook_deps`  scripts mentioned in `meta.yaml` are executed similar to `deps`
 * After this, keys in `env` dictionary is exported as `ENV` variables and `run` file if exists is executed.
-* Once run file execution is done, any `posthook_deps` CM scripts mentioned in `_cm.json` are executed similar to `deps`
+* Once run file execution is done, any `posthook_deps` scripts mentioned in `meta.yaml` are executed similar to `deps`
 * Then `postprocess` function inside customize.py is executed if present.
-* After this stage any `post_deps` CM scripts mentioned in `_cm.json` is executed.
+* After this stage any `post_deps` scripts mentioned in `meta.yaml` is executed.
 
 ** If a script is already cached, then the `preprocess`, `run file` and `postprocess` executions won't happen and only the dependencies marked as `dynamic` will be executed from `deps`, `prehook_deps`, `posthook_deps` and `postdeps`.
 
 ### Input flags
-When we run a CM script we can also pass inputs to it and any input added in `input_mapping` dictionary inside `_cm.json` gets converted to the corresponding `ENV` variable.
+When we run an MLC script we can also pass inputs to it and any input added in `input_mapping` dictionary inside `meta.yaml` gets converted to the corresponding `ENV` variable.
 
 ### Conditional execution of any `deps`, `post_deps`
 We can use `skip_if_env` dictionary inside any `deps`, `prehook_deps`, `posthook_deps` or `post_deps` to make its execution conditional
@@ -36,28 +36,28 @@ We can specify any specific version of a script using `version`. `version_max` a
 * When `version_max` is given, any version below this if present in the cache or detected in the system can be chosen. If nothing is detected `default_version` if present and if below `version_max` will be used for installation. Otherwise `version_max_usable` (additional needed input for `version_max`) will be used as `version`.
 
 ### Variations
-* Variations are used to customize CM script and each unique combination of variations uses a unique cache entry. Each variation can turn on `env` keys also any other meta including dependencies specific to it. Variations are turned on like tags but with a `_` prefix. For example, if a script is having tags `"get,myscript"`, to call the variation `"test"` inside it, we have to use tags `"get,myscript,_test"`. 
+* Variations are used to customize MLC script and each unique combination of variations uses a unique cache entry. Each variation can turn on `env` keys also any other meta including dependencies specific to it. Variations are turned on like tags but with a `_` prefix. For example, if a script is having tags `"get,myscript"`, to call the variation `"test"` inside it, we have to use tags `"get,myscript,_test"`. 
 
 #### Variation groups
 `group` is a key to map variations into a group and at any time only one variation from a group can be used in the variation tags. For example, both `cpu` and `cuda` can be two variations under the `device` group, but user can at any time use either `cpu` or `cuda` as variation tags but not both.
 
 #### Dynamic variations
 Sometimes it is difficult to add all variations needed for a script like say `batch_size` which can take many different values. To handle this case, we support dynamic variations using '#' where '#' can be dynamically replaced by any string. For example, `"_batch_size.8"` can be used as a tag to turn on the dynamic variation `"_batch_size.#"`.
 
-### ENV flow during CM script execution
+### ENV flow during MLC script execution
 
 
 * During a given script execution incoming `env` dictionary is saved `(saved_env)` and all the updates happens on a copy of it.
 * Once a script execution is over (which includes all the dependent script executions as well), newly created keys and any updated keys are merged with the `saved_env` provided the keys are mentioned in `new_env_keys`
 * Same behaviour applies to `state` dictionary.
 
 #### Special env keys
-* Any env key with a prefix `CM_TMP_*` and `CM_GIT_*` are not passed by default to any dependency. These can be force passed by adding the key(s) to the `force_env_keys` list of the concerned dependency. 
+* Any env key with a prefix `MLC_TMP_*` and `MLC_GIT_*` are not passed by default to any dependency. These can be force passed by adding the key(s) to the `force_env_keys` list of the concerned dependency. 
 * Similarly we can avoid any env key from being passed to a given dependency by adding the prefix of the key in the `clean_env_keys` list of the concerned dependency.
-* `--input` is automatically converted to `CM_INPUT` env key
-* `version` is converted to `CM_VERSION`, ``version_min` to `CM_VERSION_MIN` and `version_max` to `CM_VERSION_MAX`
-* If `env['CM_GH_TOKEN']=TOKEN_VALUE` is set then git URLs (specified by `CM_GIT_URL`) are changed to add this token.
-* If `env['CM_GIT_SSH']=yes`, then git URLs are changed to SSH from HTTPS.
+* `--input` is automatically converted to `MLC_INPUT` env key
+* `version` is converted to `MLC_VERSION`, ``version_min` to `MLC_VERSION_MIN` and `version_max` to `MLC_VERSION_MAX`
+* If `env['MLC_GH_TOKEN']=TOKEN_VALUE` is set then git URLs (specified by `MLC_GIT_URL`) are changed to add this token.
+* If `env['MLC_GIT_SSH']=yes`, then git URLs are changed to SSH from HTTPS.
 
 ### Script Meta
 #### Special keys in script meta
@@ -70,7 +70,7 @@ Sometimes it is difficult to add all variations needed for a script like say `ba
 * By default no depndencies are run for a cached entry unless `dynamic` key is set for it. 
 
 
-Please see [here](getting-started.md) for trying CM scripts.
+Please see [here](getting-started.md) for trying MLC scripts.
 
 
 

docs/requirements.txt

@@ -3,4 +3,3 @@ swagger-markdown
 mkdocs-macros-plugin
 ruamel.yaml
 slugify
-mkdocs-caseinsensitive-plugin