Skip to content

Commit 05a79cb

Browse files
CM/CMX v3.5.3: a few bug fixes (#1379)
* fixed bug with `cm info repo` ; `cmx info repo`: #1378
2 parents b607edb + 79bf743 commit 05a79cb

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+405
-190
lines changed

CONTRIBUTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,11 @@ Modify the project in your own fork and issue a pull request once you want other
1414
to take a look at what you have done and discuss the proposed changes.
1515
Ensure that cla-bot and other checks pass for your Pull requests.
1616

17-
Collective Knowledge (CK) and Collective Mind (CM)
17+
Collective Knowledge (CK), Collective Mind (CM) and Common Metadata eXchange (CMX)
1818
were created by [Grigori Fursin](https://arxiv.org/abs/2406.16791),
1919
sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons
2020
to benefit everyone. Since then, this open-source automation technology
21-
(CM/CMX, CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, etc) is being extended
21+
(CM/CMX, CM4MLOps/MLPerf automations, CM4ABTF, CM4Research, etc) is being extended
2222
as a community effort thanks to all our volunteers, collaborators
2323
and contributors listed here in alphabetical order:
2424

COPYRIGHT.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
Copyright (c) 2021-2024 MLCommons
1+
© 2021-2025 MLCommons. All Rights Reserved.
22

3-
Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone.
3+
Grigori Fursin, the cTuning foundation and OctoML donated the CK and CM projects to MLCommons to benefit everyone and continue development as a community effort.
44

55
Copyright (c) 2014-2021 cTuning foundation

HISTORY.CM.md

Lines changed: 42 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,12 @@ aimed at benefiting the broader community and fostering its ongoing development
2323
[ref4](https://github.com/mlcommons/ck/tree/master/ck),
2424
[ref5](https://github.com/mlcommons/ck-mlops).
2525

26+
Prepared MLCommons proposal for the creation of the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md),
27+
aimed at fostering community-driven support for CK and CM developments to benefit everyone.
28+
29+
* Jan, 2022: hired Arjun Suresh at OctoML to support and maintain CK1 framework and help prepare OctoML's MLPerf submissions using CK1.
30+
Meanwhile, transitioned to focusing on CM and CM-MLOps development, building upon the prototypes created in 2021.
31+
2632
* Mar 1, 2022: started developing cm-mlops: [ref](https://github.com/octoml/cm-mlops/commit/0ae94736a420dfa84f7417fc62d323303b8760c6).
2733

2834
* Mar 24, 2022: after successfully stabilizing the initial prototype of CM, donated it to MLCommons to benefit the entire community:
@@ -55,7 +61,7 @@ aimed at benefiting the broader community and fostering its ongoing development
5561
[ref3](https://github.com/octoml/cm-mlops/blob/7692240becd6397a96c3975388913ea082002e7a/CONTRIBUTIONS).
5662

5763
* May 11, 2022: After successfully prototyping CM and CM-MLOps, deprecated the CK1 framework in favor of CM.
58-
Welcomed Arjun as a maintainer and tester for CM and CM-MLOps:
64+
Transferred Arjun Suresh to the CM project as a maintainer and tester for CM and CM-MLOps:
5965
[ref](https://github.com/octoml/cm-mlops/blob/17405833665bc1e93820f9ff76deb28a0f543bdb/CONTRIBUTIONS).
6066

6167
Created a [file](https://github.com/mlcommons/ck/blob/master/cm-mlops/CHANGES.md)
@@ -75,17 +81,47 @@ aimed at benefiting the broader community and fostering its ongoing development
7581
[ref1](https://github.com/mlcommons/ck/commit/2c5d5c5c944ae5f252113c62af457c7a4c5e877a#diff-faac2c4ecfd0bfb928dafc938d3dad5651762fbb504a2544752a337294ee2573R224),
7682
[ref2](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#author-and-coordinator).
7783

78-
Welcomed Arjun Suresh as a contributor to CM: [ref](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#contributors-in-alphabetical-order).
84+
Welcomed Arjun Suresh as a contributor to CM automations for MLPerf: [ref](https://github.com/mlcommons/ck/blob/2c5d5c5c944ae5f252113c62af457c7a4c5e877a/CONTRIBUTING.md#contributors-in-alphabetical-order).
7985

8086
* From September 2022: coordinated community development of CM and CM4MLOps
81-
to [modularize and automate MLPerf](https://docs.mlcommons.org/inference)
87+
to [modularize and automate MLPerf benchmarks](https://docs.mlcommons.org/inference)
8288
and support [reproducibility initiatives at ML and Systems conferences](https://cTuning.or/ae)
83-
through the MLCommons Task Force on Automation and Reproducibility.
89+
through the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md).
90+
91+
* Directed and financed the creation of (CM) automations to streamline the MLPerf power measurement processes.
92+
93+
* Proposed to use MLPerf benchmarks for the Student Cluster Competition, led the developments
94+
and prepared a tutorial to run MLPerf inference at SCC'22 via CM: [ref](https://github.com/mlcommons/ck/blob/master/docs/tutorials/sc22-scc-mlperf.md)
95+
96+
* April 2023: departed OctoML to focus on the development of the [CK playground](https://access.cKnowledge.org) and CM automations
97+
to make Mlperf accessible to everyone. Hired Arjun Suresh to help with developments.
98+
99+
* Initiated and funded development of the [MLPerf explorer](https://github.com/ctuning/q2a-mlperf-visualizer)
100+
to improve visualization of results
101+
102+
* August 2023: organized the 1st mass-scale MLPerf community submission of 12217 inference benchmark v3.1 results
103+
out of total 13351 results (including 90% of all power results) across diverse models, software and hardware
104+
from different vendors via [open challenges](https://access.cknowledge.org/playground/?action=challenges) funded by cTuning.org :
105+
[LinkedIn article](https://www.linkedin.com/pulse/new-milestone-make-mlperf-benchmarks-accessible-everyone-fursin/)
106+
with results visualized by the [MLPerf explorer](https://github.com/ctuning/q2a-mlperf-visualizer),
107+
[CM4MLOps challenges at GitHub](https://github.com/mlcommons/cm4mlops/tree/main/challenge).
108+
109+
* February, 2024: proposed to use CM to automate [MLPerf automotive benchmark (ABTF)](https://mlcommons.org/working-groups/benchmarks/automotive/).
110+
111+
* moved my prototypes of the CM automation for ABTF to cm4abtf repo: [ref](https://github.com/mlcommons/cm4abtf/commit/f92b9f464de89a38a4bde149290dede2d94c8631)
112+
* led further CM4ABTF developments funded by cTuning.org.
84113

85114
* Starting in April 2024, began the gradual transfer of ongoing maintenance and enhancement
86115
responsibilities for CM and CM4MLOps, including MLPerf automations, to MLCommons.
87116
Welcomed Anandhu Sooraj as a maintainer and contributor to CM4MLOps with MLPerf automations.
88117

89-
For more details, please refer to this [white paper](https://arxiv.org/abs/2406.16791)
90-
and [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339).
118+
* Took a break from all development activities.
119+
120+
* July 2024: started prototyping the next generation of CM (CMX and CMX4MLOps) with simpler interfaces
121+
based on user feedback while maintaining backward compatibility.
122+
123+
* 2025: continue developing CMX and CMX4MLOPs to make it easier to run and customize MLPerf inference, training
124+
and other benchmarks across diverse models, datasets, software and hardware.
91125

126+
For more details, please refer to the [white paper](https://arxiv.org/abs/2406.16791)
127+
and the [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339).

README.md

Lines changed: 47 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,14 @@ in the most efficient and cost-effective way across diverse models, data sets, s
1818

1919
It includes the following sub-projects.
2020

21-
### Collective Mind (CM)
21+
### Collective Minds (CM)
22+
23+
The Collective Mind (CM) project, or Collective Minds, facilitates the
24+
decomposition of complex software systems into portable, reusable, and
25+
interconnected automation recipes. These recipes are developed and
26+
continuously improved by the community.
27+
28+
#### CM automation framework
2229

2330
The [Collective Mind framework](https://github.com/mlcommons/ck/tree/master/cm)
2431
is a lightweight, Python-based toolset featuring
@@ -30,12 +37,21 @@ and other applications across diverse and continuously changing models, data, so
3037
Collective Mind is continuously enhanced through public and private Git repositories
3138
with CM automation recipes and artifacts accessible via unified CM interface.
3239

33-
The CM architecture diagram is available for viewing
40+
#### CMX automation framework
41+
42+
[CMX](https://github.com/mlcommons/ck/tree/master/cmx) is the next evolution
43+
of the Collective Mind framework designed to enhance simplicity, flexibility, and extensibility of automations
44+
based on user feedback. It is backwards compatible with CM, released along with CM
45+
in the [cmind package](https://pypi.org/project/cmind/) and can serve as drop-in replacement for CM.
46+
47+
The CM/CMX architecture diagram is available for viewing
3448
[here](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png).
3549

36-
### Notable Collective Mind repositories
3750

38-
#### CM4MLOps
51+
52+
### Notable CM use cases
53+
54+
#### MLOps and MLPerf automations
3955

4056
[CM4MLOPS repository powered by CM](https://github.com/mlcommons/ck/tree/master/cm-mlops) -
4157
a collection of portable, extensible and technology-agnostic automation recipes
@@ -57,27 +73,19 @@ while keeping backward compatibility.
5773
See the [online documentation](https://docs.mlcommons.org/inference)
5874
at MLCommons to run MLPerf inference benchmarks across diverse systems using CM.
5975

60-
#### CM4ABTF
76+
#### MLCommons ABTF automation
6177

6278
[CM4ABTF repository powered by CM](https://github.com/mlcommons/cm4abtf) -
6379
a collection of portable automations and CM scripts to run the upcoming
6480
automotive MLPerf benchmark across different models, data sets, software
6581
and hardware from different vendors.
6682

67-
#### CM4MLPerf-results
83+
#### MLPerf results visualization
6884

6985
[CM4MLPerf-results powered by CM](https://github.com/mlcommons/cm4mlperf-results) -
7086
a simplified and unified representation of the past MLPerf results
7187
in the CM format for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments).
7288

73-
#### CM4Research
74-
75-
[CM4Research repository powered by CM](https://github.com/ctuning/cm4research) -
76-
a unified interface designed to streamline the preparation, execution, and reproduction of experiments in research projects.
77-
78-
79-
### Projects powered by Collective Mind
80-
8189
#### Collective Knowledge Playground
8290

8391
[Collective Knowledge Playground](https://access.cKnowledge.org) -
@@ -97,18 +105,23 @@ collaboratively enhance the efficiency and cost-effectiveness of AI systems.
97105
leveraging the Collective Mind framework to automate artifact evaluation
98106
and support reproducibility efforts at ML and systems conferences.
99107

108+
* [CM4Research repository powered by CM](https://github.com/ctuning/cm4research) -
109+
a unified interface designed to streamline the preparation, execution, and reproduction of experiments in research projects.
110+
100111

101-
## Incubator
112+
## Legacy projects
102113

103-
[CMX](https://github.com/mlcommons/ck/tree/master/cmx) - the next evolution of the Collective Mind framework,
104-
designed to enhance simplicity, flexibility, and extensibility of automations
105-
based on user feedback. Follow the project's progress [here]( https://github.com/orgs/mlcommons/projects/46 ).
114+
### CM-MLOps (now CM4MLOps)
106115

116+
You can find CM-MLOps original dev directory [here](https://github.com/mlcommons/ck/tree/master/cm-mlops).
117+
We moved it to [CM4MLOps](https://github.com/mlcommons/ck/tree/master/cm4mlops) in 2024.
118+
In 2025, we aggregate all CM and CMX automations in the [new CMX4MLOps repository](https://github.com/mlcommons/ck/tree/master/cmx4mlops).
107119

108-
## Archived projects
120+
### CK automation framework v1 and v2
109121

110-
* [CM-MLOps](https://github.com/mlcommons/ck/tree/master/cm-mlops) - now [CM4MLOps](https://github.com/mlcommons/ck/tree/master/cm4mlops)
111-
* [CK automation framework v1 and v2](https://github.com/mlcommons/ck/tree/master/ck) - now [CM](https://github.com/mlcommons/ck/tree/master/cm)
122+
You can find the original CK automation framework v1 and v2 directory [here](https://github.com/mlcommons/ck/tree/master/ck).
123+
It was deprecated for the [CM framework](https://github.com/mlcommons/ck/tree/master/cm)
124+
and later for the [CMX workflow automation framework (backwards compatible with CM)](https://github.com/mlcommons/ck/tree/master/cmx)
112125

113126

114127
## License
@@ -117,17 +130,17 @@ based on user feedback. Follow the project's progress [here]( https://github.com
117130

118131
## Copyright
119132

120-
* Copyright (c) 2021-2024 MLCommons
121-
* Copyright (c) 2014-2021 cTuning foundation
133+
Copyright (c) 2021-2024 MLCommons
134+
135+
Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone.
136+
137+
Copyright (c) 2014-2021 cTuning foundation
122138

123139
## Author
124140

125141
* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning)
126142

127-
## Citing Collective Mind and Collective Knowledge
128-
129-
If you found the CM automation framework helpful, kindly reference this article:
130-
[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].
143+
## Long-term vision
131144

132145
To learn more about the motivation behind CK and CM technology, please explore the following presentations:
133146

@@ -136,10 +149,9 @@ To learn more about the motivation behind CK and CM technology, please explore t
136149
* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]
137150
* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ]
138151

152+
## Documentation
139153

140-
## CM Documentation
141-
142-
* [Collective Mind white paper](https://arxiv.org/abs/2406.16791)
154+
* [White paper](https://arxiv.org/abs/2406.16791)
143155
* [CM/CMX architecture](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png)
144156
* [CM/CMX installation GUI](https://access.cknowledge.org/playground/?action=install)
145157
* [CM Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md)
@@ -149,13 +161,12 @@ To learn more about the motivation behind CK and CM technology, please explore t
149161
* [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials)
150162
* [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/README.md)
151163
* [CM taskforce](https://github.com/mlcommons/ck/tree/master/docs/taskforce.md)
152-
* [CK, CM and CMX history](https://github.com/mlcommons/ck/tree/master/docs/history.md)
164+
* History: [CK](https://github.com/mlcommons/ck/tree/master/docs/history.md), [CM and CM automations for MLOps and MLPerf](https://github.com/mlcommons/ck/blob/master/HISTORY.CM.md)
153165

154166

155167
### Acknowledgments
156168

157-
The open-source Collective Knowledge project (CK, CM, CM4MLOps/CM4MLPerf,
158-
CM4Research and CMX) was created by [Grigori Fursin](https://cKnowledge.org/gfursin)
169+
This open-source project was created by [Grigori Fursin](https://cKnowledge.org/gfursin)
159170
and sponsored by cTuning.org, OctoAI and HiPEAC.
160171
Grigori donated CK to MLCommons to benefit the community
161172
and to advance its development as a collaborative, community-driven effort.
@@ -164,3 +175,6 @@ We thank [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai)
164175
and [cTuning](https://cTuning.org) for supporting this project,
165176
as well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)
166177
for their feedback and contributions!
178+
179+
If you found the CM automations helpful, kindly reference this article:
180+
[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ].

cm-mlops/COPYRIGHT.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copyright Notice
2+
3+
© 2022-2025 MLCommons. All Rights Reserved.
4+
5+
Grigori Fursin, the cTuning foundation and OctoML donated the CK and CM projects to MLCommons to benefit everyone and continue development as a community effort.

cm-mlops/COPYRIGHT.txt

Lines changed: 0 additions & 5 deletions
This file was deleted.

cm/CHANGES.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
## V3.5.3
2+
- fixed `cm info repo` and `cmx info repo`: https://github.com/mlcommons/ck/issues/1378
3+
14
## V3.5.2
25
- added `-i` flag to print memory and disk use before running CM/CMX commands:
36
https://github.com/mlcommons/ck/issues/1375

cm/COPYRIGHT.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Copyright (c) 2022-2024 MLCommons
1+
Copyright (c) 2022-2025 MLCommons
22

33
Grigori Fursin, the cTuning foundation and OctoML
44
donated CM and CM4MLOps to MLCommons to benefit everyone.

0 commit comments

Comments
 (0)