V2.3.9: improved "cm init" and added support to set CLI flags to False (#1298)

- added `--min` == `--skip` to `cm init` for readability
- added `--checkout` to `cm init` to handle checkout
- added support to set CLI parameter to False if it ends with `-`, 
     i.e. `--no-cache-` -> `i['no_cache'] = False`
     `--no-cache` -> `i['no_cache'] = True`

Author: ctuning-admin

COPYRIGHT.txt

@@ -1,5 +1,5 @@
 Copyright (c) 2021-2024 MLCommons
 
-The cTuning foundation and OctoML donated this project to MLCommons to benefit everyone.
+Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone.
 
 Copyright (c) 2014-2021 cTuning foundation

README.md

@@ -55,9 +55,9 @@ CK consists of several ongoing sub-projects:
 
 ### Incubator
 
-We are preparing new projects based on user feedback:
+We are preparing new projects based on user feedback - please contact [Grigori Fursin](https://cKnowledge.org/gfursin) for more details:
 * [The next generation of CM](_incubator/cm-next-gen) *(prototyping stage)*
-* [The crowd-testing infrastructure for CM4MLOps and CM4MLPerf](_incubator/cm4mlops-testing) *(brainstorming stage)*
+* [Collaborative testing of MLPerf benchmarks](_incubator/cm4mlops-testing) *(brainstorming stage)*
 
 
 ### License
@@ -72,8 +72,7 @@ We are preparing new projects based on user feedback:
 
 ### Documentation
 
-**MLCommons is updating the CM documentation based on user feedback - please check stay tuned for more details**.
-
+* [CM installation GUI](https://access.cknowledge.org/playground/?action=install)
 * [CM Getting Started Guide and FAQ](docs/getting-started.md)
   * [Common CM interface to run MLPerf inference benchmarks](docs/mlperf/inference)
   * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](docs/tutorials/common-interface-to-reproduce-research-projects.md)

ck/COPYRIGHT.txt

@@ -1,5 +1,5 @@
 Copyright (c) 2021-2022 MLCommons
 
-cTuning foundation and OctoML donated this project to MLCommons to benefit everyone. 
+Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone. 
 
 Copyright (c) 2014-2021 cTuning foundation

cm-mlops/COPYRIGHT.txt

@@ -0,0 +1,5 @@
+Copyright (c) 2022-2023 MLCommons
+
+Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone. 
+
+Since 2024, we use a separate Github repository for further developments: https://github.com/mlcommons/cm4mlops

cm/CHANGES.md

@@ -1,3 +1,10 @@
+## V2.3.9
+   - added `--min` == `--skip` to `cm init` for readability
+   - added `--checkout` to `cm init` to handle checkout
+   - added support to set CLI parameter to False if it ends with `-`, 
+     i.e. `--no-cache-` -> `i['no_cache'] = False`
+     `--no-cache` -> `i['no_cache'] = True`
+
 ## V2.3.8
    - added `--skip` and `--url` flags to `cm init`
    - added support to pull CM repos using --url with "git@"

cm/COPYRIGHT.txt

@@ -1,2 +1,4 @@
 Copyright (c) 2022-2024 MLCommons
 
+Grigori Fursin, the cTuning foundation and OctoML 
+donated CM and CM4MLOps to MLCommons to benefit everyone. 

cm/cmind/__init__.py

@@ -2,7 +2,7 @@
 #
 # Written by Grigori Fursin
 
-__version__ = "2.3.8"
+__version__ = "2.3.9"
 
 from cmind.core import access
 from cmind.core import error

cm/cmind/cli.py

@@ -233,8 +233,12 @@ def parse(cmd):
                key = a[:j].strip()
                value = a[j+1:].strip()
             else:
-               key=a
-               value=True
+               if a.endswith('-'):
+                   key=a[:-1]
+                   value=False
+               else:
+                   key=a
+                   value=True
 
             if key.startswith('-'): key=key[1:]
             if key.startswith('-'): key=key[1:]

cm/cmind/repo/automation/core/module.py

@@ -15,7 +15,8 @@
 
 class CMInit():
     ###############################################################
-    def run(self, quiet = False, skip = False, repo_name = 'mlcommons@cm4mlops', repo_url = '', repo_branch = ''):
+    def run(self, quiet = False, skip = False, repo_name = 'mlcommons@cm4mlops', repo_url = '', 
+                  repo_branch = '', repo_checkout = ''):
         import cmind
 
         print ('Checking platform information ...')
@@ -47,6 +48,9 @@ def run(self, quiet = False, skip = False, repo_name = 'mlcommons@cm4mlops', rep
             if repo_branch !='':
                 ii['branch'] = repo_branch
 
+            if repo_checkout !='':
+                ii['checkout'] = repo_checkout
+
             rr = cmind.access(ii)
 
         return rr
@@ -228,7 +232,9 @@ def init(self, i):
           (repo) (str): main automation repository to pull ('mlcommons@cm4mlops' by default) 
           (url) (str): main automation repository to pull via url (can use git@ instead of https)
           (branch) (str): branch to use ('' by default)
+          (checkout) (str): Git checkout ('' by default)
           (skip) (bool): skip pulling main automation repository
+          (min) (bool): the same as `skip`
 
         Returns:
           (CM return dict):
@@ -242,15 +248,23 @@ def init(self, i):
         quiet = i.get('quiet', False)
 
         skip = i.get('skip', False)
+        if not skip:
+            skip = i.get('min', False)
 
         repo_name = i.get('repo', '').strip()
         repo_url = i.get('url', '').strip()
         if repo_url == '' and repo_name == '': 
             repo_name = 'mlcommons@cm4mlops'
 
         repo_branch = i.get('branch', '')
-
-        r = cm_init.run(quiet = quiet, skip = skip, repo_name = repo_name, repo_url = repo_url, repo_branch = repo_branch)
+        repo_checkout = i.get('checkout', '')
+
+        r = cm_init.run(quiet = quiet, 
+                        skip = skip, 
+                        repo_name = repo_name, 
+                        repo_url = repo_url, 
+                        repo_branch = repo_branch,
+                        repo_checkout = repo_checkout)
         if r['return']>0: return r
 
         warning = r.get('warning', '')

cm/cmind/utils.py

@@ -1792,3 +1792,35 @@ def check_if_none_false_no_off(env, key):
 
     return str(env.get(key, '')).lower() in ['none', 'false', 'no', 'off']
 
+##############################################################################
+def convert_dictionary(d, key, sub = True):
+    """
+    Grigori added to gradually clean up very complex "cm docker scrpit" implementation
+
+    Convert dictionary into flat dictionary with key prefix
+
+    Example input:
+      d = {'cfg':True, 'no-cache':True}
+      key = 'docker'
+      sub = True
+    Example output:
+      d = {'docker_cfg': True, 'docker_no_cache': True}
+
+    Args:    
+
+       d (dict): dictionary
+       key (str): key
+
+    Returns:
+       True if str(env.get(key, '')).lower() in ['false', 'no', 'off']:
+    """
+
+    dd = {}
+
+    for k in d:
+        kk = k.replace('-', '_') if sub else k
+
+        dd[key + '_' + kk] = d[k]
+
+    return dd
+

docs/installation.md

@@ -17,21 +17,25 @@
 
 </details>
 
-
-***Check our new [online installation GUI](https://access.cknowledge.org/playground/?action=install)***.
+**Last revision of this document: September 29, 2024**
 
 
 # CM installation
 
 
 
-
 MLCommons Collective Mind framework requires minimal dependencies to run on any platform: `python 3+, pip, git, git-lfs, wget`.
 However, most CM automation recipes shared by the community and MLCommons require Python 3.7+ .
 
 ***By default, CM will pull Git repositories and cache installations and downloaded files in your `$HOME/CM` directory (Linux/MacOS). 
    You can change it to any another directory using the `CM_REPOS` environment variable, for example `export CM_REPOS=/scratch/CM`.***
 
+***We suggest you not to install `cm4mlops` package via PIP since you can't control installation of the CM framework and repositories 
+   and it doesn't handle CM errors properly - use a newer version of `cm init` after installing `cmind` package as described below.***
+
+***Feel free to use the [online installation GUI](https://access.cknowledge.org/playground/?action=install)***.
+
+
 Here are typical installation procedures across different operating systems:
 
 * [Ubuntu, Debian](#ubuntu-debian)
@@ -173,6 +177,7 @@ python -m pip install cmind
 *We plan to provide a self-sustained package in the future to simplify CM installation on Windows.*
 
 
+
 # CM CLI testing
 
 If the installation is successful, you can run the CM CLI as follows:
@@ -205,6 +210,24 @@ Reporting issues and ideas:        https://github.com/mlcommons/ck/issues
 Joining the open MLPerf workgroup: https://cKnowledge.org/mlcommons-taskforce
 ```
 
+# CM init
+
+Use the following command to test CM system dependencies (git, wget, curl, etc):
+
+```bash
+cm init
+```
+
+Note that it will also install stable `cm4mlops` repository with the automation recipes
+for MLOps and MLPerf.
+
+You can skip installation of this repository and use the standard CM command to pull this repo
+as follows:
+```bash
+cm init --min
+cm pull repo mlcommons@cm4mlops
+```
+
 # CUDA installation
 
 If you plan to use CUDA for your experiments, please follow [this guide](installation-cuda.md)