Skip to content
This repository was archived by the owner on Aug 26, 2022. It is now read-only.

Commit ecf16ed

Browse files
hyunwoongkohyunwoongko
authored
Merge branch 'master' into kevin
2 parents 0cec28d + 3d7e792 commit ecf16ed

32 files changed

+3805
-209
lines changed

‎docs/CONFIGURATION/activation_checkpointing.html

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,12 @@
2222

2323

2424

25-
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
26-
<link rel="stylesheet" href="../_static/sphinx-book-theme.css?digest=84ace793992934648b4de8eed757e5a2" type="text/css" />
25+
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
26+
<link rel="stylesheet" type="text/css" href="../_static/sphinx-book-theme.css?digest=84ace793992934648b4de8eed757e5a2" />
2727

2828
<link rel="preload" as="script" href="../_static/js/index.be7d3bbb2ef33a8344ce.js">
2929

30-
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
30+
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
3131
<script src="../_static/jquery.js"></script>
3232
<script src="../_static/underscore.js"></script>
3333
<script src="../_static/doctools.js"></script>
@@ -72,7 +72,7 @@ <h1 class="site-logo" id="site-title">OSLO documentation</h1>
7272
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
7373
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
7474
<div class="bd-toc-item active">
75-
<p class="caption">
75+
<p aria-level="2" class="caption" role="heading">
7676
<span class="caption-text">
7777
CONFIGURATION
7878
</span>
@@ -94,6 +94,38 @@ <h1 class="site-logo" id="site-title">OSLO documentation</h1>
9494
</a>
9595
</li>
9696
</ul>
97+
<p aria-level="2" class="caption" role="heading">
98+
<span class="caption-text">
99+
TUTORIALS
100+
</span>
101+
</p>
102+
<ul class="nav bd-sidenav">
103+
<li class="toctree-l1">
104+
<a class="reference internal" href="../TUTORIALS/tensor_model_parallelism.html">
105+
Tensor Model Parallelism Tutorial
106+
</a>
107+
</li>
108+
<li class="toctree-l1">
109+
<a class="reference internal" href="../TUTORIALS/tensor_model_data_parallelism.html">
110+
Tensor Model + Data Parallelism Tutorial
111+
</a>
112+
</li>
113+
<li class="toctree-l1">
114+
<a class="reference internal" href="../TUTORIALS/tensor_model_zero_data_parallelism.html">
115+
Tensor Model + ZeRO Data Parallelism Tutorial
116+
</a>
117+
</li>
118+
<li class="toctree-l1">
119+
<a class="reference internal" href="../TUTORIALS/activation_checkpointing.html">
120+
Activation Checkpointing Tutorial
121+
</a>
122+
</li>
123+
<li class="toctree-l1">
124+
<a class="reference internal" href="../TUTORIALS/kernel_fusion.html">
125+
Kernel Fusion Tutorial
126+
</a>
127+
</li>
128+
</ul>
97129

98130
</div>
99131
</nav> <!-- To handle the deprecated key -->
@@ -274,7 +306,7 @@ <h2> Contents </h2>
274306
<div>
275307

276308
<div class="section" id="activation-checkpointing">
277-
<h1>Activation Checkpointing<a class="headerlink" href="#activation-checkpointing" title="Permalink to this headline"></a></h1>
309+
<h1>Activation Checkpointing<a class="headerlink" href="#activation-checkpointing" title="Permalink to this headline"></a></h1>
278310
<p>You can specify your own activation checkpointing related configuration under <code class="docutils literal notranslate"><span class="pre">activation_checkpointing</span></code> like:</p>
279311
<div class="highlight-json notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
280312
<span class="nt">&quot;activation_checkpointing&quot;</span><span class="p">:</span> <span class="p">{</span>
@@ -287,23 +319,23 @@ <h1>Activation Checkpointing<a class="headerlink" href="#activation-checkpointin
287319
</pre></div>
288320
</div>
289321
<div class="section" id="enable-bool">
290-
<h2>1. enable: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#enable-bool" title="Permalink to this headline"></a></h2>
322+
<h2>1. enable: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#enable-bool" title="Permalink to this headline"></a></h2>
291323
<ul class="simple">
292324
<li><p>type: bool</p></li>
293325
<li><p>default: False</p></li>
294326
</ul>
295327
<p>Enable activation checkpointing.</p>
296328
</div>
297329
<div class="section" id="cpu-checkpointing-bool">
298-
<h2>2. cpu_checkpointing: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#cpu-checkpointing-bool" title="Permalink to this headline"></a></h2>
330+
<h2>2. cpu_checkpointing: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#cpu-checkpointing-bool" title="Permalink to this headline"></a></h2>
299331
<ul class="simple">
300332
<li><p>type: bool</p></li>
301333
<li><p>default: False</p></li>
302334
</ul>
303335
<p>Enable cpu checkpointing.</p>
304336
</div>
305337
<div class="section" id="partitioned-checkpointing-bool">
306-
<h2>3. partitioned_checkpointing: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#partitioned-checkpointing-bool" title="Permalink to this headline"></a></h2>
338+
<h2>3. partitioned_checkpointing: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#partitioned-checkpointing-bool" title="Permalink to this headline"></a></h2>
307339
<ul class="simple">
308340
<li><p>type: bool</p></li>
309341
<li><p>default: False</p></li>
@@ -312,7 +344,7 @@ <h2>3. partitioned_checkpointing: <code class="docutils literal notranslate"><sp
312344
<p>Note that this is only available when you are using tensor model parallelism.</p>
313345
</div>
314346
<div class="section" id="contiguous-checkpointing-bool">
315-
<h2>4. contiguous_checkpointing: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#contiguous-checkpointing-bool" title="Permalink to this headline"></a></h2>
347+
<h2>4. contiguous_checkpointing: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#contiguous-checkpointing-bool" title="Permalink to this headline"></a></h2>
316348
<p>Enable contiguous checkpointing.</p>
317349
<p>Note that this is only available when you are using partitioned checkpointing.</p>
318350
</div>

‎docs/CONFIGURATION/kernel_fusion.html

Lines changed: 48 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,18 +22,19 @@
2222

2323

2424

25-
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
26-
<link rel="stylesheet" href="../_static/sphinx-book-theme.css?digest=84ace793992934648b4de8eed757e5a2" type="text/css" />
25+
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
26+
<link rel="stylesheet" type="text/css" href="../_static/sphinx-book-theme.css?digest=84ace793992934648b4de8eed757e5a2" />
2727

2828
<link rel="preload" as="script" href="../_static/js/index.be7d3bbb2ef33a8344ce.js">
2929

30-
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
30+
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
3131
<script src="../_static/jquery.js"></script>
3232
<script src="../_static/underscore.js"></script>
3333
<script src="../_static/doctools.js"></script>
3434
<script src="../_static/sphinx-book-theme.9d8b4a8b9bb19db25eeaddc40d639ba2.js"></script>
3535
<link rel="index" title="Index" href="../genindex.html" />
3636
<link rel="search" title="Search" href="../search.html" />
37+
<link rel="next" title="Tensor Model Parallelism Tutorial" href="../TUTORIALS/tensor_model_parallelism.html" />
3738
<link rel="prev" title="Activation Checkpointing" href="activation_checkpointing.html" />
3839
<meta name="viewport" content="width=device-width, initial-scale=1" />
3940
<meta name="docsearch:language" content="None">
@@ -71,7 +72,7 @@ <h1 class="site-logo" id="site-title">OSLO documentation</h1>
7172
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
7273
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
7374
<div class="bd-toc-item active">
74-
<p class="caption">
75+
<p aria-level="2" class="caption" role="heading">
7576
<span class="caption-text">
7677
CONFIGURATION
7778
</span>
@@ -93,6 +94,38 @@ <h1 class="site-logo" id="site-title">OSLO documentation</h1>
9394
</a>
9495
</li>
9596
</ul>
97+
<p aria-level="2" class="caption" role="heading">
98+
<span class="caption-text">
99+
TUTORIALS
100+
</span>
101+
</p>
102+
<ul class="nav bd-sidenav">
103+
<li class="toctree-l1">
104+
<a class="reference internal" href="../TUTORIALS/tensor_model_parallelism.html">
105+
Tensor Model Parallelism Tutorial
106+
</a>
107+
</li>
108+
<li class="toctree-l1">
109+
<a class="reference internal" href="../TUTORIALS/tensor_model_data_parallelism.html">
110+
Tensor Model + Data Parallelism Tutorial
111+
</a>
112+
</li>
113+
<li class="toctree-l1">
114+
<a class="reference internal" href="../TUTORIALS/tensor_model_zero_data_parallelism.html">
115+
Tensor Model + ZeRO Data Parallelism Tutorial
116+
</a>
117+
</li>
118+
<li class="toctree-l1">
119+
<a class="reference internal" href="../TUTORIALS/activation_checkpointing.html">
120+
Activation Checkpointing Tutorial
121+
</a>
122+
</li>
123+
<li class="toctree-l1">
124+
<a class="reference internal" href="../TUTORIALS/kernel_fusion.html">
125+
Kernel Fusion Tutorial
126+
</a>
127+
</li>
128+
</ul>
96129

97130
</div>
98131
</nav> <!-- To handle the deprecated key -->
@@ -253,7 +286,7 @@ <h2> Contents </h2>
253286
<div>
254287

255288
<div class="section" id="kernel-fusion">
256-
<h1>Kernel Fusion<a class="headerlink" href="#kernel-fusion" title="Permalink to this headline"></a></h1>
289+
<h1>Kernel Fusion<a class="headerlink" href="#kernel-fusion" title="Permalink to this headline"></a></h1>
257290
<p>You can specify your own kernel fusion related configuration under <code class="docutils literal notranslate"><span class="pre">kernel_fusion</span></code> like:</p>
258291
<div class="highlight-json notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
259292
<span class="nt">&quot;kernel_fusion&quot;</span><span class="p">:</span> <span class="p">{</span>
@@ -265,23 +298,23 @@ <h1>Kernel Fusion<a class="headerlink" href="#kernel-fusion" title="Permalink to
265298
</pre></div>
266299
</div>
267300
<div class="section" id="enable-bool">
268-
<h2>1. enable: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#enable-bool" title="Permalink to this headline"></a></h2>
301+
<h2>1. enable: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#enable-bool" title="Permalink to this headline"></a></h2>
269302
<ul class="simple">
270303
<li><p>type: bool</p></li>
271304
<li><p>default: False</p></li>
272305
</ul>
273306
<p>Enable kernel fusion.</p>
274307
</div>
275308
<div class="section" id="memory-efficient-fusion-bool">
276-
<h2>2. memory_efficient_fusion: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#memory-efficient-fusion-bool" title="Permalink to this headline"></a></h2>
309+
<h2>2. memory_efficient_fusion: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#memory-efficient-fusion-bool" title="Permalink to this headline"></a></h2>
277310
<ul class="simple">
278311
<li><p>type: bool</p></li>
279312
<li><p>default: False</p></li>
280313
</ul>
281314
<p>Enable memory efficient fusion.</p>
282315
</div>
283316
<div class="section" id="custom-cuda-kernels-list">
284-
<h2>3. custom_cuda_kernels: <code class="docutils literal notranslate"><span class="pre">list</span></code><a class="headerlink" href="#custom-cuda-kernels-list" title="Permalink to this headline"></a></h2>
317+
<h2>3. custom_cuda_kernels: <code class="docutils literal notranslate"><span class="pre">list</span></code><a class="headerlink" href="#custom-cuda-kernels-list" title="Permalink to this headline"></a></h2>
285318
<ul class="simple">
286319
<li><p>type: list</p></li>
287320
<li><p>default: []</p></li>
@@ -308,6 +341,13 @@ <h2>3. custom_cuda_kernels: <code class="docutils literal notranslate"><span cla
308341
<p class="prev-next-title">Activation Checkpointing</p>
309342
</div>
310343
</a>
344+
<a class='right-next' id="next-link" href="../TUTORIALS/tensor_model_parallelism.html" title="next page">
345+
<div class="prev-next-info">
346+
<p class="prev-next-subtitle">next</p>
347+
<p class="prev-next-title">Tensor Model Parallelism Tutorial</p>
348+
</div>
349+
<i class="fas fa-angle-right"></i>
350+
</a>
311351
</div>
312352

313353
</div>

‎docs/CONFIGURATION/model_parallelism.html

Lines changed: 42 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,12 @@
2222

2323

2424

25-
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
26-
<link rel="stylesheet" href="../_static/sphinx-book-theme.css?digest=84ace793992934648b4de8eed757e5a2" type="text/css" />
25+
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
26+
<link rel="stylesheet" type="text/css" href="../_static/sphinx-book-theme.css?digest=84ace793992934648b4de8eed757e5a2" />
2727

2828
<link rel="preload" as="script" href="../_static/js/index.be7d3bbb2ef33a8344ce.js">
2929

30-
<script id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
30+
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
3131
<script src="../_static/jquery.js"></script>
3232
<script src="../_static/underscore.js"></script>
3333
<script src="../_static/doctools.js"></script>
@@ -72,7 +72,7 @@ <h1 class="site-logo" id="site-title">OSLO documentation</h1>
7272
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
7373
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
7474
<div class="bd-toc-item active">
75-
<p class="caption">
75+
<p aria-level="2" class="caption" role="heading">
7676
<span class="caption-text">
7777
CONFIGURATION
7878
</span>
@@ -94,6 +94,38 @@ <h1 class="site-logo" id="site-title">OSLO documentation</h1>
9494
</a>
9595
</li>
9696
</ul>
97+
<p aria-level="2" class="caption" role="heading">
98+
<span class="caption-text">
99+
TUTORIALS
100+
</span>
101+
</p>
102+
<ul class="nav bd-sidenav">
103+
<li class="toctree-l1">
104+
<a class="reference internal" href="../TUTORIALS/tensor_model_parallelism.html">
105+
Tensor Model Parallelism Tutorial
106+
</a>
107+
</li>
108+
<li class="toctree-l1">
109+
<a class="reference internal" href="../TUTORIALS/tensor_model_data_parallelism.html">
110+
Tensor Model + Data Parallelism Tutorial
111+
</a>
112+
</li>
113+
<li class="toctree-l1">
114+
<a class="reference internal" href="../TUTORIALS/tensor_model_zero_data_parallelism.html">
115+
Tensor Model + ZeRO Data Parallelism Tutorial
116+
</a>
117+
</li>
118+
<li class="toctree-l1">
119+
<a class="reference internal" href="../TUTORIALS/activation_checkpointing.html">
120+
Activation Checkpointing Tutorial
121+
</a>
122+
</li>
123+
<li class="toctree-l1">
124+
<a class="reference internal" href="../TUTORIALS/kernel_fusion.html">
125+
Kernel Fusion Tutorial
126+
</a>
127+
</li>
128+
</ul>
97129

98130
</div>
99131
</nav> <!-- To handle the deprecated key -->
@@ -248,7 +280,7 @@ <h2> Contents </h2>
248280
<div>
249281

250282
<div class="section" id="model-parallelism">
251-
<h1>Model Parallelism<a class="headerlink" href="#model-parallelism" title="Permalink to this headline"></a></h1>
283+
<h1>Model Parallelism<a class="headerlink" href="#model-parallelism" title="Permalink to this headline"></a></h1>
252284
<p>You can specify your own model parallelism related configuration under <code class="docutils literal notranslate"><span class="pre">model_parallelism</span></code> like:</p>
253285
<div class="highlight-json notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
254286
<span class="nt">&quot;model_parallelism&quot;</span><span class="p">:</span> <span class="p">{</span>
@@ -259,15 +291,15 @@ <h1>Model Parallelism<a class="headerlink" href="#model-parallelism" title="Perm
259291
</pre></div>
260292
</div>
261293
<div class="section" id="enable-bool">
262-
<h2>1. enable: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#enable-bool" title="Permalink to this headline"></a></h2>
294+
<h2>1. enable: <code class="docutils literal notranslate"><span class="pre">bool</span></code><a class="headerlink" href="#enable-bool" title="Permalink to this headline"></a></h2>
263295
<ul class="simple">
264296
<li><p>type: bool</p></li>
265297
<li><p>default: False</p></li>
266298
</ul>
267299
<p>Enable model parallelism.</p>
268300
</div>
269301
<div class="section" id="tensor-parallel-size-int">
270-
<h2>2. tensor_parallel_size: <code class="docutils literal notranslate"><span class="pre">int</span></code><a class="headerlink" href="#tensor-parallel-size-int" title="Permalink to this headline"></a></h2>
302+
<h2>2. tensor_parallel_size: <code class="docutils literal notranslate"><span class="pre">int</span></code><a class="headerlink" href="#tensor-parallel-size-int" title="Permalink to this headline"></a></h2>
271303
<ul class="simple">
272304
<li><p>type: int</p></li>
273305
<li><p>default: 1</p></li>
@@ -281,12 +313,14 @@ <h2>2. tensor_parallel_size: <code class="docutils literal notranslate"><span cl
281313
<li><p><code class="docutils literal notranslate"><span class="pre">tensor_parallel_size</span></code> must be positive number.</p></li>
282314
</ul>
283315
<div class="section" id="supported-models">
284-
<h3>SUPPORTED MODELS<a class="headerlink" href="#supported-models" title="Permalink to this headline"></a></h3>
316+
<h3>SUPPORTED MODELS<a class="headerlink" href="#supported-models" title="Permalink to this headline"></a></h3>
285317
<p>The following list includes currently supported models.</p>
286318
<ul class="simple">
287319
<li><p>Albert</p></li>
288320
<li><p>Bert</p></li>
289321
<li><p>Bart</p></li>
322+
<li><p>Blenderbot</p></li>
323+
<li><p>BlenderbotSmall</p></li>
290324
<li><p>T5</p></li>
291325
<li><p>GPT2</p></li>
292326
<li><p>GPTNeo</p></li>

0 commit comments

Comments
 (0)