Preliminary pipeline docs (#67)

Author: lu-ohai

docs/source/index.rst

@@ -56,6 +56,7 @@ Oracle Accelerated Data Science SDK (ADS)
    user_guide/big_data_service/index
    user_guide/jobs/index
    user_guide/logs/logs
+   user_guide/pipeline/index
    user_guide/secrets/index
 
 .. toctree::

docs/source/user_guide/cli/opctl/configure.rst

@@ -3,15 +3,15 @@ CLI Configuration
 
 **Prerequisite**
 
-* You have completed :doc:`ADS CLI installation <../quickstart>` 
+* You have completed :doc:`ADS CLI installation <../quickstart>`
 
 
 Setup default values for different options while running ``OCI Data Sciecne Jobs`` or ``OCI DataFlow``. By setting defaults, you can avoid inputing compartment ocid, project ocid, etc.
 
-To setup configuration run - 
+To setup configuration run -
 
 .. code-block:: shell
-  
+
   ads opctl configure
 
 This will prompt you to setup default ADS CLI configurations for each OCI profile defined in your OCI config. By default, all the files are generated in the ``~/.ads_ops`` folder.
@@ -20,8 +20,8 @@ This will prompt you to setup default ADS CLI configurations for each OCI profil
 
 ``~/.ads_ops/config.ini`` will contain OCI profile defaults and conda pack related information. For example:
 
-.. code-block:: 
-    
+.. code-block::
+
     [OCI]
     oci_config = ~/.oci/config
     oci_profile = ANOTHERPROF
@@ -30,7 +30,7 @@ This will prompt you to setup default ADS CLI configurations for each OCI profil
     conda_pack_folder = </local/path/for/saving/condapack>
     conda_pack_os_prefix = oci://my-bucket@mynamespace/conda_environments/
 
-``~/.ads_ops/ml_job_config.ini`` will contain defaults for running ``Data Science Job``. Defaults are set for each profile listed in your oci config file. Here is a sample - 
+``~/.ads_ops/ml_job_config.ini`` will contain defaults for running ``Data Science Job``. Defaults are set for each profile listed in your oci config file. Here is a sample -
 
 .. code-block::
 
@@ -53,7 +53,7 @@ This will prompt you to setup default ADS CLI configurations for each OCI profil
     block_storage_size_in_GBs = 50
 
 
-``~/.ads_ops/dataflow_config.ini`` will contain defaults for running ``Data Science Job``. Defaults are set for each profile listed in your oci config file. Here is a sample - 
+``~/.ads_ops/dataflow_config.ini`` will contain defaults for running ``Data Science Job``. Defaults are set for each profile listed in your oci config file. Here is a sample -
 
 .. code-block::
 
@@ -66,3 +66,30 @@ This will prompt you to setup default ADS CLI configurations for each OCI profil
     num_executors = 3
     spark_version = 3.0.2
     archive_bucket = oci://mybucket@mytenancy/dataflow/archive
+
+``~/.ads_ops/ml_pipeline.ini`` will contain defaults for running ``Data Science Pipeline``. Defaults are set for each profile listed in your oci config file. Here is a sample -
+
+.. code-block::
+
+    [DEFAULT]
+    compartment_id = oci.xxxx.<compartment_ocid>
+    project_id = oci.xxxx.<project_ocid>
+
+    [ANOTHERPROF]
+    compartment_id = oci.xxxx.<compartment_ocid>
+    project_id = oci.xxxx.<project_ocid>
+
+
+``~/.ads_ops/local_backend.ini`` will contain defaults for running jobs and pipeline steps locally. While local operations do not involve connections to OCI services, default
+configurations are still set for each profile listed in your oci config file for consistency. Here is a sample -
+
+.. code-block::
+
+    [DEFAULT]
+    max_parallel_containers = 4
+    pipeline_status_poll_interval_seconds = 5
+
+
+    [ANOTHERPROF]
+    max_parallel_containers = 4
+    pipeline_status_poll_interval_seconds = 5

docs/source/user_guide/cli/opctl/local-development-setup.rst

@@ -4,18 +4,18 @@ Local Development Environment Setup
 
 **Prerequisite**
 
-* You have completed :doc:`ADS CLI installation <../quickstart>` 
-* You have completed :doc:`Configuaration <configure>` 
+* You have completed :doc:`ADS CLI installation <../quickstart>`
+* You have completed :doc:`Configuaration <configure>`
 
-Setup up your workstation for development and testing your code locally before you submit it as a OCI Data Science Job. This section will guide you on how to setup environment for - 
+Setup up your workstation for development and testing your code locally before you submit it as a OCI Data Science Job. This section will guide you on how to setup environment for -
 
 * Building an OCI Data Science compatible conda environments on your workstation or CICD pipeline and publishing to object storage
 * Developing and testing code with a conda environment that is compatible with OCI Data Science Notebooks and OCI Data Science Jobs
 * Developing and testing code for running Bring Your Own Container (BYOC) jobs.
 
 **Note**
 
-* In this version you cannot directly access the Service provided conda environments from ADS CLI, but you can publish a service provided conda pack from an OCI Data Science Notebook session to your object storage bucket and then use the CLI to access the published version. 
+* In this version you cannot directly access the Service provided conda environments from ADS CLI, but you can publish a service provided conda pack from an OCI Data Science Notebook session to your object storage bucket and then use the CLI to access the published version.
 
 .. toctree::
   :hidden:
@@ -25,5 +25,5 @@ Setup up your workstation for development and testing your code locally before y
   localdev/vscode
   localdev/condapack
   localdev/jobs
-
-
+  localdev/local_jobs
+  localdev/local_pipelines

docs/source/user_guide/cli/opctl/localdev/local_jobs.rst

@@ -0,0 +1,75 @@
++++++++++++++++++++
+Local Job Execution
++++++++++++++++++++
+
+Your job can be executed in a local container to facilitate development and troubleshooting.
+
+-------------
+Prerequisites
+-------------
+
+1. :doc:`Install ADS CLI<../../quickstart>`
+2. Build a container image.
+    - :doc:`Build Development Container Image<./jobs_container_image>` and :doc:`install a conda environment<./condapack>`
+    - :doc:`Build Your Own Container (BYOC)<./jobs>`
+
+------------
+Restrictions
+------------
+
+When running locally, your job is subject to the following restrictions:
+  - The job must use API Key auth. Resource Principal auth is not supported in a local container. See https://docs.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm
+  - You can only use conda environment published to your own Object Storage bucket. See :doc:`Working with Conda packs<./condapack>`
+  - Your job files must be present on your local machine.
+  - Any network calls must be reachable by your local machine. (i.e. Your job cannot connect to an endpoint that is only reachable within the job's subnet.)
+  - Your local machine meets the hardware requirements of your job.
+
+----------------
+Running your Job
+----------------
+
+Using a conda environment
+=========================
+
+This example below demonstrates how to run a local job using an installed conda environment:
+
+.. code-block:: shell
+
+  ads opctl run --backend local --conda-slug myconda_p38_cpu_v1 --source-folder /path/to/my/job/files/ --entrypoint bin/my_script.py --cmd-args "--some-arg" --env-var "MY_VAR=12345"
+
+Parameter explanation:
+  - ``--backend local``: Run the job locally in a docker container.
+  - ``--conda-slug myconda_p38_cpu_v1``: Use the ``myconda_p38_cpu_v1`` conda environment. Note that you must install this conda environment locally first.
+    The local conda environment directory will be automatically mounted into the container and activated before the entrypoint is executed.
+  - ``--source-folder /path/to/my/job/files/``: The local directory containing your job files. This directory is mounted into the container as a volume.
+  - ``--entrypoint bin/my_script.py``: Set the container entrypoint to ``bin/my_script.py``. Note that this path is relative to the path specified with the ``--source-folder`` parameter.
+  - ``--cmd-args "--some-arg"``: Pass ``--some-arg`` to the container entrypoint.
+  - ``--env-var "MY_VAR=12345": Define envrionment variable ``MY_VAR`` with value ``12345``.
+
+Using a custom image
+====================
+
+This example below demonstrates how to run a local job using a custom container image:
+
+.. code-block:: shell
+
+  ads opctl run --backend local --image my_image --entrypoint /path/to/my/binary --command my_cmd --env-var "MY_VAR=12345"
+
+Parameter explanation:
+  - ``--backend local``: Run the job locally in a docker container.
+  - ``--image my_image``: Use the custom container image named ``my_image``.
+  - ``--entrypoint /path/to/my/binary``: Set the container entrypoint to ``/path/to/my/binary``. Note that this path is within the container image.
+  - ``--command my_cmd``: Set the container command to ``my_cmd``.
+  - ``--env-var "MY_VAR=12345": Define envrionment variable ``MY_VAR`` with value ``12345``.
+
+Viewing container output
+========================
+When the container is running, you can use the ``docker logs`` command to view its output. See https://docs.docker.com/engine/reference/commandline/logs/
+
+Alternatively, you can use the ``--debug`` parameter to print the container stdout/stderr messages to your shell. Note that Python buffers output by default, so you may see output written
+to the shell in bursts. If you want to see output displayed in real-time, specify ``--env-var PYTHONUNBUFFERED=1``.
+
+.. code-block:: shell
+
+  ads opctl run --backend local --conda-slug myconda_p38_cpu_v1 --source-folder /path/to/my/job/files/ --entrypoint my_script.py --env-var "PYTHONUNBUFFERED=1" --debug
+

docs/source/user_guide/cli/opctl/localdev/local_pipelines.rst

@@ -0,0 +1,153 @@
+++++++++++++++++++++++++
+Local Pipeline Execution
+++++++++++++++++++++++++
+
+Your pipeline can be executed locally to facilitate development and troubleshooting. Each pipeline step is executed in its own local container.
+
+-------------
+Prerequisites
+-------------
+
+1. :doc:`Install ADS CLI<../../quickstart>`
+2. :doc:`Build Development Container Image<./jobs_container_image>` and :doc:`install a conda environment<./condapack>`
+
+------------
+Restrictions
+------------
+
+Your pipeline steps are subject to the :doc:`same restrictions as local jobs<./local_jobs>`.
+
+They are also subject to these additional restrictions:
+
+  - Pipeline steps must be of kind ``customScript``.
+  - Custom container images are not yet supported. You must use the development container image with a conda environment.
+
+---------------------------------------
+Configuring Local Pipeline Orchestrator
+---------------------------------------
+
+Use ``ads opctl configure``. Refer to the ``local_backend.ini`` description in the configuration :doc:`instructions<../configure>`.
+
+Most importantly, ``max_parallel_containers`` controls how many pipeline steps may be executed in parallel on your machine. Your pipeline DAG may allow multiple steps to be executed in parallel,
+but your local machine may not have enough cpu cores / memory to effectively run them all simultaneously.
+
+---------------------
+Running your Pipeline
+---------------------
+
+Local pipeline execution requires you to define your pipeline in a yaml file. Refer to the YAML examples :doc:`here<../../../pipeline/examples>`.
+
+Then, invoke the following command to run your pipeline.
+
+.. code-block:: shell
+
+  ads opctl run --backend local --file my_pipeline.yaml --source-folder /path/to/my/pipeline/step/files
+
+Parameter explanation:
+  - ``--backend local``: Run the pipeline locally using docker containers.
+  - ``--file my_pipeline.yaml``: The yaml file defining your pipeline.
+  - ``--source-folder /path/to/my/pipeline/step/files``: The local directory containing the files used by your pipeline steps. This directory is mounted into the container as a volume.
+    Defaults to the current working directory if no value is provided.
+
+Source folder and relative paths
+================================
+If your pipeline step runtimes are of type ``script`` or ``notebook``, the paths in your yaml files must be relative to the ``--source-folder``.
+
+Pipeline steps using a runtime of type ``python`` are able to define their own working directory that will be mounted into the step's container instead.
+
+For example, suppose your yaml file looked like this:
+
+.. code-block:: yaml
+
+  kind: pipeline
+  spec:
+    displayName: example
+    dag:
+    - (step_1, step_2) >> step_3
+    stepDetails:
+    - kind: customScript
+      spec:
+        description: A step running a notebook
+        name: step_1
+        runtime:
+          kind: runtime
+          spec:
+            conda:
+              slug: myconda_p38_cpu_v1
+              type: service
+            notebookEncoding: utf-8
+            notebookPathURI: step_1_files/my-notebook.ipynb
+            type: notebook
+    - kind: customScript
+      spec:
+        description: A step running a shell script
+        name: step_2
+        runtime:
+          kind: runtime
+          spec:
+            conda:
+              slug: myconda_p38_cpu_v1
+              type: service
+            scriptPathURI: step_2_files/my-script.sh
+            type: script
+    - kind: customScript
+      spec:
+        description: A step running a python script
+        name: step_3
+        runtime:
+          kind: runtime
+          spec:
+            conda:
+              slug: myconda_p38_cpu_v1
+              type: service
+            workingDir: /step_3/custom/working/dir
+            scriptPathURI: my-python.py
+            type: python
+  type: pipeline
+
+And suppose the pipeline is executed locally with the following command:
+
+.. code-block:: shell
+
+  ads opctl run --backend local --file my_pipeline.yaml --source-folder /my/files
+
+``step_1`` uses a ``notebook`` runtime. The container for ``step_1`` will mount the ``/my/files`` directory into the container. The ``/my/files/step_1_files/my-notebook.ipynb`` notebook file
+will be converted into a python script and executed in the container.
+
+``step_2`` uses a ``script`` runtime. The container for ``step_2`` will mount the ``/my/files`` directory into the container. The ``/my/files/step_2_files/my-script.sh`` shell script will
+be executed in the container.
+
+``step_3`` uses a ``python`` runtime. Instead of mounting the ``/my/files`` directory specified by ``--source-folder``, the ``/step_3/custom/working/dir`` directory will be mounted into the
+container. The ``/step_3/custom/working/dir/my-python.py`` script will be executed in the container.
+
+Viewing container output and orchestration messages
+===================================================
+When a container is running, you can use the ``docker logs`` command to view its output. See https://docs.docker.com/engine/reference/commandline/logs/
+
+Alternatively, you can use the ``--debug`` parameter to print each container's stdout/stderr messages to your shell. Note that Python buffers output by default, so you may see output written
+to the shell in bursts. If you want to see output displayed in real-time for a particular step, specify a non-zero value for the ``PYTHONUNBUFFERED`` environment variable in your step's runtime
+specification. For example:
+
+.. code-block:: yaml
+
+  - kind: customScript
+    spec:
+      description: A step running a shell script
+      name: step_1
+      runtime:
+        kind: runtime
+        spec:
+          conda:
+            slug: myconda_p38_cpu_v1
+            type: service
+          scriptPathURI: my-script.sh
+          env:
+            PYTHONUNBUFFERED: 1
+        type: script
+
+
+Pipeline steps can run in parallel. You may want your pipeline steps to prefix their log output to easily distinguish which lines of output are coming from which step.
+
+When the ``--debug`` parameter is specified, the CLI will also output pipeline orchestration messages. These include messages about which steps are being started and a summary of each
+step's result when the pipeline finishes execution.
+

docs/source/user_guide/jobs/_template/runtime_types.rst

@@ -0,0 +1,4 @@
+* ``GitPythonRuntime`` allows you to run source code from a Git repository, see :ref:`Run from Git <job_run_git>`.
+* ``NotebookRuntime`` allows you to run a JupyterLab Python notebook, see :ref:`Run a Notebook <job_run_a_notebook>`.
+* ``PythonRuntime`` allows you to run Python code with additional options, including setting a working directory, adding Python paths, and copying output files, see :ref:`Run a ZIP file or folder <job_run_zip>`.
+* ``ScriptRuntime`` allows you to run Python, Bash, and Java scripts from a single source file (``.zip`` or ``.tar.gz``) or code directory, see :ref:`Run a Script <job_run_script>` and :ref:`Run a ZIP file or folder <job_run_zip>`.

docs/source/user_guide/jobs/data_science_job.rst

@@ -85,10 +85,7 @@ Runtime
 
 A job can have different types of *runtime* depending on the source code you want to run:
 
-* ``ScriptRuntime`` allows you to run Python, Bash, and Java scripts from a single source file (``.zip`` or ``.tar.gz``) or code directory, see `Run a Script <run_script.html>`__ and `Run a ZIP file or folder <run_zip.html>`__.
-* ``PythonRuntime`` allows you to run Python code with additional options, including setting a working directory, adding python paths, and copying output files, see `Run a ZIP file or folder <run_zip.html>`__.
-* ``NotebookRuntime`` allows you to run a JupyterLab Python notebook, see `Run a Notebook <run_notebook.html>`__.
-* ``GitPythonRuntime`` allows you to run source code from a Git repository, see `Run from Git <run_git.html>`__.
+.. include:: _template/runtime_types.rst
 
 All of these runtime options allow you to configure a `Data Science Conda Environment <https://docs.oracle.com/en-us/iaas/data-science/using/conda_understand_environments.htm>`__ for running your code. For example, to define a python script as a job runtime with a TensorFlow conda environment you could use:
 

docs/source/user_guide/jobs/index.rst

@@ -18,3 +18,4 @@ Oracle Cloud Infrastructure (OCI) Data Science jobs enable you to define and run
    run_zip
    ../cli/opctl/_template/jobs
    ../cli/opctl/_template/monitoring
+   ../cli/opctl/localdev/local_jobs

docs/source/user_guide/jobs/run_git.rst

@@ -1,3 +1,5 @@
+.. _job_run_git:
+
 Run a Git Repo
 **************
 

docs/source/user_guide/jobs/run_notebook.rst

@@ -1,3 +1,5 @@
+.. _job_run_a_notebook:
+
 Run a Notebook
 **************