Skip to content

Commit 15dbef3

Browse files
qinxuyewjsi
authored andcommitted
Add installation, instructions and getting involved in README, add local cluster docs (#51)
* add install, instructions in readme, add local cluster to install docs
1 parent 1028eeb commit 15dbef3

File tree

2 files changed

+166
-4
lines changed

2 files changed

+166
-4
lines changed

README.rst

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,24 @@ Mars
55

66
Mars is a tensor-based unified framework for large-scale data computation.
77

8+
Installation
9+
------------
10+
11+
Mars is easy to install by
12+
13+
.. code-block:: bash
14+
15+
pip install pymars
16+
17+
The distributed version can be installed by
18+
19+
.. code-block:: bash
20+
21+
pip install 'pymars[distributed]'
22+
23+
For now, distributed version is only available on Linux and Mac OS.
24+
25+
826
Mars tensor
927
-----------
1028

@@ -56,6 +74,102 @@ Mars can scale in to a single machine, and scale out to a cluster with thousands
5674
Both the local and distributed version share the same piece of code,
5775
it's fairly simple to migrate from a single machine to a cluster due to the increase of data.
5876

77+
Running on a single machine including thread-based scheduling,
78+
local cluster scheduling which bundles the whole distributed components.
79+
Mars is also easy to scale out to a cluster by starting different components of
80+
mars distributed runtime on different machines in the cluster.
81+
82+
Threaded
83+
````````
84+
85+
``execute`` method will by default run on the thread-based scheduler on a single machine.
86+
87+
.. code-block:: python
88+
89+
import mars.tensor as mt
90+
91+
a = mt.ones(10, 10)
92+
a.execute()
93+
94+
Users can create a session explicitly.
95+
96+
.. code-block:: python
97+
98+
from mars.session import new_session
99+
100+
session = new_session()
101+
session.run(a + 1)
102+
(a * 2).execute(session=session)
103+
104+
# session will be released when out of with statement
105+
with new_session() as session2:
106+
session2.run(a / 3)
107+
108+
109+
Local cluster
110+
`````````````
111+
112+
Users can start the local cluster bundled with the distributed runtime on a single machine.
113+
Local cluster mode requires mars distributed version.
114+
115+
.. code-block:: python
116+
117+
from mars.deploy.local import new_cluster
118+
119+
# cluster will create a session and set it as default
120+
cluster = new_cluster()
121+
122+
# run on the local cluster
123+
(a + 1).execute()
124+
125+
# create a session explicitly by specifying the cluster's endpoint
126+
session = new_session(cluster.endpoint)
127+
session.run(a * 3)
128+
129+
130+
Distributed
131+
```````````
132+
133+
After installing the distributed version on every node in the cluster,
134+
A node can be selected as scheduler and another as web service,
135+
leaving other nodes as workers. The scheduler can be started with the following command:
136+
137+
.. code-block:: bash
138+
139+
mars-scheduler -a <scheduler_ip> -p <scheduler_port>
140+
141+
Web service can be started with the following command:
142+
143+
.. code-block:: bash
144+
145+
mars-web -a <web_ip> -s <scheduler_ip> --ui-port <ui_port_exposed_to_user>
146+
147+
Workers can be started with the following command:
148+
149+
.. code-block:: bash
150+
151+
mars-worker -a <worker_ip> -p <worker_port> -s <scheduler_ip>
152+
153+
After all mars processes are started, users can run
154+
155+
.. code-block:: python
156+
157+
sess = new_session('http://<web_ip>:<ui_port>')
158+
a = mt.ones((2000, 2000), chunks=200)
159+
b = mt.inner(a, a)
160+
sess.run(b)
161+
162+
163+
Getting involved
164+
----------------
165+
166+
- Join the mailing list: send an email to `mars-dev@googlegroups.com`_.
167+
- Please report bugs by submitting a `GitHub issue`_.
168+
- Submit contributions using `pull requests`_.
169+
170+
Thank you in advance for your contributions!
171+
172+
59173
.. |Build| image:: https://img.shields.io/travis/mars-project/mars.svg?style=flat-square
60174
:target: https://travis-ci.org/mars-project/mars
61175
.. |Coverage| image:: https://img.shields.io/coveralls/github/mars-project/mars.svg?style=flat-square
@@ -67,3 +181,6 @@ it's fairly simple to migrate from a single machine to a cluster due to the incr
67181
.. |License| image:: https://img.shields.io/pypi/l/pymars.svg?style=flat-square
68182
:target: https://github.com/mars-project/mars/blob/master/LICENSE
69183
.. |Implementation| image:: https://img.shields.io/pypi/implementation/pymars.svg?style=flat-square
184+
.. _`mars-dev@googlegroups.com`: https://groups.google.com/forum/#!forum/mars-dev
185+
.. _`GitHub issue`: https://github.com/mars-project/mars/issues
186+
.. _`pull requests`: https://github.com/mars-project/mars/pulls

docs/source/install.rst

Lines changed: 49 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
1-
Standalone install
2-
==================
1+
Standalone mode
2+
===============
3+
4+
Threaded
5+
--------
6+
37
You can install Mars via pip:
48

59
.. code-block:: bash
@@ -12,10 +16,51 @@ After installation, you can simply open a Python console and run
1216
1317
import mars.tensor as mt
1418
from mars.session import new_session
15-
sess = new_session()
19+
1620
a = mt.ones((5, 5), chunks=3)
1721
b = a * 4
18-
sess.run(b)
22+
# if there isn't a local session,
23+
# execute will create a default one first
24+
b.execute()
25+
26+
# or create a session explicitly
27+
sess = new_session()
28+
sess.run(b) # run b
29+
30+
31+
Local cluster
32+
-------------
33+
34+
Users can start the distributed runtime of Mars on a single machine.
35+
First, install Mars distributed by run
36+
37+
.. code-block:: bash
38+
39+
pip install 'pymars[distributed]'
40+
41+
For now, local cluster mode can only run on Linux and Mac OS.
42+
43+
Then start a local cluster by run
44+
45+
.. code-block:: python
46+
47+
from mars.deploy.local import new_cluster
48+
49+
cluster = new_cluster()
50+
51+
# new cluster will start a session and set it as default one
52+
# execute will then run in the local cluster
53+
a = mt.random.rand(10, 10)
54+
a.dot(a.T).execute()
55+
56+
# cluster.session is the session created
57+
cluster.session.run(a + 1)
58+
59+
# users can also create a session explicitly
60+
# cluster.endpoint needs to be passed to new_session
61+
session2 = new_session(cluster.endpoint)
62+
session2.run(a * 2)
63+
1964
2065
Run on Clusters
2166
===============

0 commit comments

Comments
 (0)