diff --git a/docs/assets/images/guides/jobs/create_new_job.png b/docs/assets/images/guides/jobs/create_new_job.png index 96ed71366..8a0d9df0e 100644 Binary files a/docs/assets/images/guides/jobs/create_new_job.png and b/docs/assets/images/guides/jobs/create_new_job.png differ diff --git a/docs/assets/images/guides/jobs/jobs_overview.png b/docs/assets/images/guides/jobs/jobs_overview.png index 5db37be88..39074082c 100644 Binary files a/docs/assets/images/guides/jobs/jobs_overview.png and b/docs/assets/images/guides/jobs/jobs_overview.png differ diff --git a/docs/assets/images/guides/jobs/ray_application_logs.png b/docs/assets/images/guides/jobs/ray_application_logs.png new file mode 100644 index 000000000..77e924aa4 Binary files /dev/null and b/docs/assets/images/guides/jobs/ray_application_logs.png differ diff --git a/docs/assets/images/guides/jobs/ray_dashboard.png b/docs/assets/images/guides/jobs/ray_dashboard.png new file mode 100644 index 000000000..0ab8db7ac Binary files /dev/null and b/docs/assets/images/guides/jobs/ray_dashboard.png differ diff --git a/docs/assets/images/guides/jobs/ray_execution.png b/docs/assets/images/guides/jobs/ray_execution.png new file mode 100644 index 000000000..3d189232e Binary files /dev/null and b/docs/assets/images/guides/jobs/ray_execution.png differ diff --git a/docs/assets/images/guides/jobs/ray_runtime_env_and_additional_files.png b/docs/assets/images/guides/jobs/ray_runtime_env_and_additional_files.png new file mode 100644 index 000000000..fef03a27e Binary files /dev/null and b/docs/assets/images/guides/jobs/ray_runtime_env_and_additional_files.png differ diff --git a/docs/assets/images/guides/jobs/resource_configuration_ray.png b/docs/assets/images/guides/jobs/resource_configuration_ray.png new file mode 100644 index 000000000..ac03bc323 Binary files /dev/null and b/docs/assets/images/guides/jobs/resource_configuration_ray.png differ diff --git a/docs/assets/images/guides/jobs/start_ray_job.png b/docs/assets/images/guides/jobs/start_ray_job.png new file mode 100644 index 000000000..84ef2d3ad Binary files /dev/null and b/docs/assets/images/guides/jobs/start_ray_job.png differ diff --git a/docs/assets/images/guides/jobs/upload_ray_script.png b/docs/assets/images/guides/jobs/upload_ray_script.png new file mode 100644 index 000000000..2c46b47ac Binary files /dev/null and b/docs/assets/images/guides/jobs/upload_ray_script.png differ diff --git a/docs/assets/images/guides/jupyter/environment_list.png b/docs/assets/images/guides/jupyter/environment_list.png new file mode 100644 index 000000000..78e5c04bd Binary files /dev/null and b/docs/assets/images/guides/jupyter/environment_list.png differ diff --git a/docs/assets/images/guides/jupyter/jupyter_overview.png b/docs/assets/images/guides/jupyter/jupyter_overview.png new file mode 100644 index 000000000..140fced4d Binary files /dev/null and b/docs/assets/images/guides/jupyter/jupyter_overview.png differ diff --git a/docs/assets/images/guides/jupyter/ray_jupyter_notebook_session.png b/docs/assets/images/guides/jupyter/ray_jupyter_notebook_session.png new file mode 100644 index 000000000..1b12ff70c Binary files /dev/null and b/docs/assets/images/guides/jupyter/ray_jupyter_notebook_session.png differ diff --git a/docs/assets/images/guides/jupyter/ray_kernel.png b/docs/assets/images/guides/jupyter/ray_kernel.png new file mode 100644 index 000000000..6c7c00469 Binary files /dev/null and b/docs/assets/images/guides/jupyter/ray_kernel.png differ diff --git a/docs/assets/images/guides/jupyter/ray_resource_and_compute.png b/docs/assets/images/guides/jupyter/ray_resource_and_compute.png new file mode 100644 index 000000000..775233c6c Binary files /dev/null and b/docs/assets/images/guides/jupyter/ray_resource_and_compute.png differ diff --git a/docs/user_guides/projects/jobs/ray_job.md b/docs/user_guides/projects/jobs/ray_job.md new file mode 100644 index 000000000..99312f4a2 --- /dev/null +++ b/docs/user_guides/projects/jobs/ray_job.md @@ -0,0 +1,210 @@ +--- +description: Documentation on how to configure and execute a Ray job on Hopsworks. +--- + +# How To Run A Ray Job + +## Introduction + +All members of a project in Hopsworks can launch the following types of applications through a project's Jobs service: + +- Python (*Hopsworks Enterprise only*) +- Apache Spark +- Ray + +Launching a job of any type is very similar process, what mostly differs between job types is +the various configuration parameters each job type comes with. Hopsworks support scheduling to run jobs on a regular basis, +e.g backfilling a Feature Group by running your feature engineering pipeline nightly. Scheduling can be done both through the UI and the python API, +checkout [our Scheduling guide](schedule_job.md). + +!!!warning "Enable Ray" + + Support for Ray needs to be explicitly enabled by adding the following option in the `values.yaml` file for the deployment: + + ```yaml + global: + ray: + enabled: true + ``` + +## UI + +### Step 1: Jobs overview + +The image below shows the Jobs overview page in Hopsworks and is accessed by clicking `Jobs` in the sidebar. + +

+

+ Jobs overview +
Jobs overview
+
+

+ +### Step 2: Create new job dialog + +Click `New Job` and the following dialog will appear. + +

+

+ Create new job dialog +
Create new job dialog
+
+

+ +### Step 3: Set the job type + +By default, the dialog will create a Spark job. Make sure `RAY` is chosen. + +### Step 4: Set the script + +Next step is to select the program to run. You can either select `From project`, if the file was previously uploaded to Hopsworks, or `Upload new file` which lets you select a file from your local filesystem as demonstrated below. After that set the name for the job. By default, the job name is the same as the file name, but you can customize it here. + +

+

+ Configure program +
Configure program
+
+

+ +### Step 5 (optional): Advanced configuration + +Resource allocation for the Driver and Workers can be configured. + +!!! notice "Using the resources in the Ray script" + The resource configurations describe the cluster that will be provisioned when launching the Ray job. User can still + provide extra configurations in the job script using `ScalingConfig`, i.e. `ScalingConfig(num_workers=4, trainer_resources={"CPU": 1}, use_gpu=True)`. + +* `Driver memory`: Memory in MBs to allocate for Driver + +* `Driver virtual cores`: Number of cores to allocate for the Driver + +* `Worker memory`: Memory in MBs to allocate for each worker + +* `Worker cores`: Number of cores to allocate for each worker + +* `Min workers`: Minimum number of workers to start with + +* `Max workers`: Maximum number of workers to scale up to + + +

+

+ Resource configuration 
+for Ray Job +
Resource configuration for the Ray Job
+
+

+ +Runtime environment and Additional files required for the Ray job can also be provided. + +* `Runtime Environment (Optional)`: A [runtime environment](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments) describes the dependencies required for the Ray job including files, packages, environment variables, and more. This is useful when you need to install specific packages and set environment variables for this particular Ray job. It should be provided as a YAML file. You can select the file from the project or upload a new one. + +* `Additional files`: List of other files required for the Ray job. These files will be placed in `/srv/hops/ray/job`. + +

+

+ Runtime 
+environment and additional files +
Runtime configuration and additional files for Ray job
+
+

+ +### Step 6: Execute the job + +Now click the `Run` button to start the execution of the job, and then click on `Executions` to see the list of all executions. + +

+

+ Start job execution +
Start job execution
+
+

+ +## Ray Dashboard +When the Ray job is running, you can access the Ray dashboard to monitor the job. The Ray dashboard is accessible from the +`Executions` page. Please note that the Ray dashboard is only available when the job execution is running. In the Ray Dashboard, +you can monitor the resources used by the job, the number of workers, logs, and the tasks that are running. +

+

+ Access Ray Dashboard +
Access Ray Dashboard
+
+

+ +### Step 9: Application logs +Once the execution is finished, you can click on `Logs` to see the full logs for execution. + +

+

+ Access Ray logs +
Access Ray job execution logs
+
+

+ +## Code + +### Step 1: Upload the Ray script + +This snippet assumes the Ray program is in the current working directory and named `ray_job.py`. If the file is already in the project, you can skip this step. + +It will upload the jar to the `Resources` dataset in your project. + +```python + +import hopsworks + +project = hopsworks.login() + +dataset_api = project.get_dataset_api() + +uploaded_file_path = dataset_api.upload("ray_job.py", "Resources") + +``` + + +### Step 2: Create Ray job + +In this snippet we get the `JobsApi` object to get the default job configuration for a `RAY` job, set the python script to run and create the `Job` object. + +```python + +jobs_api = project.get_jobs_api() + +ray_config = jobs_api.get_configuration("RAY") + +ray_config['appPath'] = uploaded_file_path +ray_config['environmentName'] = "ray-training-pipeline" +ray_config['driverCores'] = 2 +ray_config['driverMemory'] = 2048 +ray_config['workerCores'] = 2 +ray_config['workerMemory'] = 4096 +ray_config['minWorkers'] = 1 +ray_config['maxWorkers'] = 4 + +job = jobs_api.create_job("ray_job", ray_config) + +``` + +### Step 3: Execute the job + +In this snippet we execute the job synchronously, that is wait until it reaches a terminal state, and then download and print the logs. + +```python + +execution = job.run(await_termination=True) + +out, err = execution.download_logs() + +f_out = open(out, "r") +print(f_out.read()) + +f_err = open(err, "r") +print(f_err.read()) + +``` + +### API Reference + +[Jobs](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/jobs/) + +[Executions](https://docs.hopsworks.ai/hopsworks-api/{{{ hopsworks_version }}}/generated/api/executions/) \ No newline at end of file diff --git a/docs/user_guides/projects/jupyter/ray_notebook.md b/docs/user_guides/projects/jupyter/ray_notebook.md new file mode 100644 index 000000000..d6d4eae3e --- /dev/null +++ b/docs/user_guides/projects/jupyter/ray_notebook.md @@ -0,0 +1,142 @@ +--- +description: Documentation on how to run Ray applications on Jupyter on Hopsworks. +--- +# How To Run A Ray Notebook + +### Introduction + +Jupyter is provided as a service in Hopsworks, providing the same user experience and features as if run on your laptop. + +* Supports JupyterLab and the classic Jupyter front-end +* Configured with Python3, PySpark and Ray kernels + +!!!warning "Enable Ray" + + Support for Ray needs to be explicitly enabled by adding the following option in the `values.yaml` file for the deployment: + + ```yaml + global: + ray: + enabled: true + ``` + +## Step 1: Jupyter dashboard + +The image below shows the Jupyter service page in Hopsworks and is accessed by clicking `Jupyter` in the sidebar. + +

+

+ Jupyter dashboard in Hopsworks +
Jupyter dashboard in Hopsworks
+
+

+ +From this page, you can configure various options and settings to start Jupyter with as described in the sections below. + +## Step 2 (Optional): Configure Ray + +Next step is to configure the Ray cluster configuration that will be created when you start Ray session later in +Jupyter. Click `edit configuration` to get to the configuration page and select `Ray`. + +### Resource and compute + +Resource allocation for the Driver and Workers can be configured. + +!!! notice "Using the resources in the Ray script" + The resource configurations describe the cluster that will be provisioned when launching the Ray job. User can still + provide extra configurations in the job script using `ScalingConfig`, i.e. `ScalingConfig(num_workers=4, trainer_resources={"CPU": 1}, use_gpu=True)`. + +* `Driver memory`: Memory in MBs to allocate for Driver + +* `Driver virtual cores`: Number of cores to allocate for the Driver + +* `Worker memory`: Memory in MBs to allocate for each worker + +* `Worker cores`: Number of cores to allocate for each worker + +* `Min workers`: Minimum number of workers to start with + +* `Max workers`: Maximum number of workers to scale up to + +

+

+ Resource configuration for 
+the Ray kernels +
Resource configuration for the Ray kernels
+
+

+ +Runtime environment and Additional files required for the Ray job can also be provided. + +* `Runtime Environment (Optional)`: A [runtime environment](https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments) describes the dependencies required for the Ray job including files, packages, environment variables, and more. This is useful when you need to install specific packages and set environment variables for this particular Ray job. It should be provided as a YAML file. You can select the file from the project or upload a new one. + + +* `Additional files`: List of other files required for the Ray job. These files will be placed in `/srv/hops/ray/job`. + +

+

+ Runtime 
+environment and additional files +
Runtime configuration and additional files for Ray jupyter session
+
+

+ + +Click `Save` to save the new configuration. + +## Step 3 (Optional): Configure max runtime and root path + +Before starting the server there are two additional configurations that can be set next to the `Run Jupyter` button. + +The runtime of the Jupyter instance can be configured, this is useful to ensure that idle instances will not be hanging around and keep allocating resources. If a limited runtime is not desirable, this can be disabled by setting `no limit`. + +

+

+ Configure maximum runtime +
Configure maximum runtime
+
+

+ +The root path from which to start the Jupyter instance can be configured. By default it starts by setting the `/Jupyter` folder as the root. + +

+

+ Configure root folder +
Configure root folder
+
+

+ +## Step 4: Select the environment +Hopsworks provides a variety of environments to run Jupyter notebooks. Select the environment you want to use by clicking on the dropdown menu. +In order to be able to run a Ray notebook, you need to select the environment that has the Ray kernel installed. +Environment with Ray kernel have a `Ray Enabled` label next to them. + +## Step 5: Start Jupyter + +Start the Jupyter instance by clicking the `Run Jupyter` button. + +## Running Ray code in Jupyter +Once the Jupyter instance is started, you can create a new notebook by clicking on the `New` button and selecting +`Ray` kernel. You can now write and run Ray code in the notebook. When you first run a cell with Ray code, a Ray session will be started and you can monitor the resources used by the job in the Ray dashboard. + +

+

+ Ray Kernel +
Ray Kernel
+
+

+ +## Step 6: Access Ray Dashboard + +When you start a Ray session in Jupyter, a new application will appear in the Jupyter page. +The notebook name from which the session was started is displayed. You can access the Ray UI by clicking on the `Ray Dashboard` and a new +tab will be opened. The Ray dashboard is only available while the Ray kernel is running. +You can kill the Ray session to free up resources by shutting down the kernel in Jupyter. +In the Ray Dashboard, you can monitor the resources used by code you are running, the number of workers, logs, and the tasks that are running. + +

+

+ Access Ray Dashboard +
Access Ray Dashboard for Jupyter Ray session
+
+

\ No newline at end of file diff --git a/docs/user_guides/projects/jupyter/spark_notebook.md b/docs/user_guides/projects/jupyter/spark_notebook.md index 3209d8ee7..c358bee61 100644 --- a/docs/user_guides/projects/jupyter/spark_notebook.md +++ b/docs/user_guides/projects/jupyter/spark_notebook.md @@ -5,7 +5,7 @@ Jupyter is provided as a service in Hopsworks, providing the same user experience and features as if run on your laptop. * Supports JupyterLab and the classic Jupyter front-end -* Configured with Python and PySpark kernels +* Configured with Python3, PySpark and Ray kernels ## Step 1: Jupyter dashboard @@ -34,7 +34,8 @@ You can easily refer to the green ticks as to what kernels are available in whic ## Step 3 (Optional): Configure spark properties -Next step is to configure the Spark properties to be used in Jupyter, Click `edit configuration` to get to the configuration page and select `Spark`. +Next step is to configure the Ray properties to be used in Jupyter, Click `edit configuration` to get to the +configuration page and select `Ray`. ### Resource and compute diff --git a/mkdocs.yml b/mkdocs.yml index c94fd7d7d..485e5791a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -147,11 +147,13 @@ nav: - Jupyter: - Run Python Notebook: user_guides/projects/jupyter/python_notebook.md - Run PySpark Notebook: user_guides/projects/jupyter/spark_notebook.md + - Run Ray Notebook: user_guides/projects/jupyter/ray_notebook.md - Remote Filesystem Driver: user_guides/projects/jupyter/remote_filesystem_driver.md - Jobs: - Run PySpark Job: user_guides/projects/jobs/pyspark_job.md - Run Spark Job: user_guides/projects/jobs/spark_job.md - Run Python Job: user_guides/projects/jobs/python_job.md + - Run Ray Job: user_guides/projects/jobs/ray_job.md - Run Jupyter Notebook Job: user_guides/projects/jobs/notebook_job.md - Scheduling: user_guides/projects/jobs/schedule_job.md - Airflow: user_guides/projects/airflow/airflow.md