diff --git a/docs/admin/aiinitiatives/org/scheduling-rules.md b/docs/admin/aiinitiatives/org/scheduling-rules.md index 72f7d1d912..5ef988b7cb 100644 --- a/docs/admin/aiinitiatives/org/scheduling-rules.md +++ b/docs/admin/aiinitiatives/org/scheduling-rules.md @@ -3,8 +3,8 @@ This article explains the procedure of configuring and managing Scheduling rules There are 3 types of rules: * **Workload time limit** - This rule limits the duration of a workload run time. Workload run time is calculated as the total time in which the workload was in status “Running“. -* **Idle GPU time limit** - This rule limits the total GPU time of a workload. Workload idle time is counted since the first time the workload was in status “Running“ and the GPU was idle. - For fractional workloads, workloads running on a MIG slice, multi GPU or multi-node workloads, each GPU idle second is calculated as follows: __ Request for CPU memory (burstable workload) | +| CPU compute utilization | The utilization of all CPU compute resources on this node (percentage) | +| CPU memory utilization | The utilization of all CPU memory resources on this node (percentage) | +| Used swap CPU memory | The amount of CPU memory (in GB or MB) used for GPU swap memory (* future) | +| Pod(s) | List of pods running on this node, click the field to view details (described below in this article) | + +### GPU devices for node + +Click one of the values in the GPU devices column, to view the list of GPU devices and their parameters. + +| Column | Description | +| :---- | :---- | +| Index | The GPU index, read from the GPU hardware. The same index is used when accessing the GPU directly | +| Used memory | The amount of memory used by pods and drivers using the GPU (in GB or MB) | +| Compute utilization | The portion of time the GPU is being used by applications (percentage) | +| Memory utilization | The portion of the GPU memory that is being used by applications (percentage) | +| Idle time | The elapsed time since the GPU was used (i.e. the GPU is being idle for ‘Idle time’) | + +### Pods associated with node + +Click one of the values in the Pod(s) column, to view the list of pods and their parameters. + +Note + +This column is only viewable if your role in the Run:ai platform gives you read access to workloads, even if you are allowed to view workloads, you can only view the workloads within your allowed scope. This means, there might be more pods running on this node than appear in the list your are viewing. + +| Column | Description | +| :---- | :---- | +| Pod | The Kubernetes name of the pod. Usually name of the pod is made of the name of the parent workload if there is one, and an index for unique for that pod instance within the workload | +| Status | The state of the pod. In steady state this should be Running and the amount of time the pod is running | +| Project | The Run:ai project name the pod belongs to. Clicking this field takes you to the Projects table filtered by this project name | +| Workload | The workload name the pod belongs to. Clicking this field takes you to the Workloads table filtered by this workload name | +| Image | The full path of the image used by the main container of this pod | +| Creation time | The pod’s creation date and time | + +### Customizing the table view + +* Filter - Click ADD FILTER, select the column to filter by, and enter the filter values +* Search - Click SEARCH and type the value to search by +* Sort - Click each column header to sort by +* Column selection - Click COLUMNS and select the columns to display in the table +* Download table - Click MORE and then Click Download as CSV +* Show/Hide details - Click to view additional information on the selected row + +### Show/Hide details + +Click a row in the Nodes table and then click the Show details button at the upper right side of the action bar. The details screen appears, presenting the following metrics graphs: + +* GPU utilization + Per GPU graph and an average of all GPUs graph, all on the same chart, along an adjustable period allows you to see the trends of all GPUs compute utilization (percentage of GPU compute) in this node. +* GPU memory utilization + Per GPU graph and an average of all GPUs graph, all on the same chart, along an adjustable period allows you to see the trends of all GPUs memory usage (percentage of the GPU memory) in this node. +* CPU compute utilization + The average of all CPUs’ cores compute utilization graph, along an adjustable period allows you to see the trends of CPU compute utilization (percentage of CPU compute) in this node. +* CPU memory utilization + The utilization of all CPUs memory in a single graph, along an adjustable period allows you to see the trends of CPU memory utilization (percentage of CPU memory) in this node. +* CPU memory usage + The usage of all CPUs memory in a single graph, along an adjustable period allows you to see the trends of CPU memory usage (in GB or MB of CPU memory) in this node. + +* For GPUs charts - Click the GPU legend on the right-hand side of the chart, to activate or deactivate any of the GPU lines. +* You can click the date picker to change the presented period +* You can use your mouse to mark a sub-period in the graph for zooming in, and use the ‘Reset zoom’ button to go back to the preset period +* Changes in the period affect all graphs on this screen. + +## Using API + +Go to the [Nodes](https://app.run.ai/api/docs#tag/Nodes) API reference to view the available actions + diff --git a/mkdocs.yml b/mkdocs.yml index d5d9869fe1..60949fb6cf 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -216,8 +216,8 @@ nav: - 'Projects' : 'admin/aiinitiatives/org/projects.md' - 'Departments' : 'admin/aiinitiatives/org/departments.md' - 'Scheduling Rules' : 'admin/aiinitiatives/org/scheduling-rules.md' - # - 'Managing your resources' : - # - 'Nodes' : 'admin/aiinitiatives/resources/nodes.md' + - 'Managing your resources' : + - 'Nodes' : 'admin/aiinitiatives/resources/nodes.md' # - 'Node Pools' : 'admin/aiinitiatives/resources/node-pools.md' - 'Review your performance' : # - 'Overview' : 'admin/admin-ui-setup/overview.md'