Skip to content

Commit a76cd37

Browse files
authored
Merge branch 'main' into odsc-58209/ntbad
2 parents 6b79c60 + 7ba9fa8 commit a76cd37

File tree

23 files changed

+103923
-3
lines changed

23 files changed

+103923
-3
lines changed

ads/opctl/operator/lowcode/anomaly/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ The operator will run in your local environment without requiring any additional
5858

5959
## 4. Running anomaly detection on the local container
6060

61-
To run the anomaly detection detection operator within a local container, follow these steps:
61+
To run the anomaly detection operator within a local container, follow these steps:
6262

6363
Use the command below to build the anomaly detection container.
6464

@@ -106,7 +106,7 @@ ads operator run -f ~/anomaly/anomaly.yaml --backend-config ~/anomaly/backend_op
106106

107107
## 5. Running anomaly detection in the Data Science job within container runtime
108108

109-
To execute the anomaly detection detection operator within a Data Science job using container runtime, please follow the steps outlined below:
109+
To execute the anomaly detection operator within a Data Science job using container runtime, please follow the steps outlined below:
110110

111111
You can use the following command to build the anomaly detection container. This step can be skipped if you have already done this for running the operator within a local container.
112112

@@ -155,7 +155,7 @@ ads opctl watch <OCID>
155155

156156
## 6. Running anomaly detection in the Data Science job within conda runtime
157157

158-
To execute the anomaly detection detection operator within a Data Science job using conda runtime, please follow the steps outlined below:
158+
To execute the anomaly detection operator within a Data Science job using conda runtime, please follow the steps outlined below:
159159

160160
You can use the following command to build the anomaly detection conda environment.
161161

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
type: recommender
2+
version: v1
3+
conda_type: service
4+
name: Recommender Operator
5+
gpu: no
6+
keywords:
7+
- Recommender
8+
backends:
9+
- job
10+
- operator.local
11+
description: |
12+
Recommender Systems are designed to suggest relevant items, products, or content to users based on their
13+
preferences and behaviors. These systems are widely used in various industries such as e-commerce, entertainment,
14+
and social media to enhance user experience by providing personalized recommendations. They help in increasing user
15+
engagement, satisfaction, and sales by predicting what users might like or need based on their past interactions
16+
and the preferences of similar users.
Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Recommender Operator
2+
3+
Recommender Systems are designed to suggest relevant items, products, or content to users based on their preferences and behaviors. These systems are widely used in various industries such as e-commerce, entertainment, and social media to enhance user experience by providing personalized recommendations. They help in increasing user engagement, satisfaction, and sales by predicting what users might like or need based on their past interactions and the preferences of similar users.
4+
5+
6+
Below are the steps to configure and run the Recommender Operator on different resources.
7+
8+
## 1. Prerequisites
9+
10+
Follow the [CLI Configuration](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/opctl/configure.html) steps from the ADS documentation. This step is mandatory as it sets up default values for different options while running the Recommender Operator on OCI Data Science jobs or OCI Data Flow applications. If you have previously done this and used a flexible shape, make sure to adjust `ml_job_config.ini` with shape config details and `docker_registry` information.
11+
12+
- ocpus = 1
13+
- memory_in_gbs = 16
14+
- docker_registry = `<iad.ocir.io/namespace/>`
15+
16+
## 2. Generating configs
17+
18+
To generate starter configs, run the command below. This will create a list of YAML configs and place them in the `output` folder.
19+
20+
```bash
21+
ads operator init -t recommender --overwrite --output ~/recommender/
22+
```
23+
24+
The most important files expected to be generated are:
25+
26+
- `recommender.yaml`: Contains recommender related configuration.
27+
- `backend_operator_local_python_config.yaml`: This includes a local backend configuration for running recommender in a local environment. The environment should be set up manually before running the operator.
28+
- `backend_operator_local_container_config.yaml`: This includes a local backend configuration for running recommender within a local container. The container should be built before running the operator. Please refer to the instructions below for details on how to accomplish this.
29+
- `backend_job_container_config.yaml`: Contains Data Science job-related config to run recommender in a Data Science job within a container (BYOC) runtime. The container should be built and published before running the operator. Please refer to the instructions below for details on how to accomplish this.
30+
- `backend_job_python_config.yaml`: Contains Data Science job-related config to run recommender in a Data Science job within a conda runtime. The conda should be built and published before running the operator.
31+
32+
All generated configurations should be ready to use without the need for any additional adjustments. However, they are provided as starter kit configurations that can be customized as needed.
33+
34+
## 3. Running recommender on the local conda environment
35+
36+
To run recommender locally, create and activate a new conda environment (`ads-recommender`). Install all the required libraries listed in the `environment.yaml` file.
37+
38+
```yaml
39+
- report-creator
40+
- cerberus
41+
- "git+https://github.com/oracle/accelerated-data-science.git@feature/recommender#egg=oracle-ads"
42+
```
43+
44+
Please review the previously generated `recommender.yaml` file using the `init` command, and make any necessary adjustments to the input and output file locations. By default, it assumes that the files should be located in the same folder from which the `init` command was executed.
45+
46+
Use the command below to verify the recommender config.
47+
48+
```bash
49+
ads operator verify -f ~/recommender/recommender.yaml
50+
```
51+
52+
Use the following command to run the recommender within the `ads-recommender` conda environment.
53+
54+
```bash
55+
ads operator run -f ~/recommender/recommender.yaml -b local
56+
```
57+
58+
The operator will run in your local environment without requiring any additional modifications.
59+
60+
## 4. Running recommender on the local container
61+
62+
To run the recommender operator within a local container, follow these steps:
63+
64+
Use the command below to build the recommender container.
65+
66+
```bash
67+
ads operator build-image -t recommender
68+
```
69+
70+
This will create a new `recommender:v1` image, with `/etc/operator` as the designated working directory within the container.
71+
72+
73+
Check the `backend_operator_local_container_config.yaml` config file. By default, it should have a `volume` section with the `.oci` configs folder mounted.
74+
75+
```yaml
76+
volume:
77+
- "/Users/<user>/.oci:/root/.oci"
78+
```
79+
80+
Mounting the OCI configs folder is only required if an OCI Object Storage bucket will be used to store the input recommender data or output recommender result. The input/output folders can also be mounted to the container.
81+
82+
```yaml
83+
volume:
84+
- /Users/<user>/.oci:/root/.oci
85+
- /Users/<user>/recommender/data:/etc/operator/data
86+
- /Users/<user>/recommender/result:/etc/operator/result
87+
```
88+
89+
The full config can look like:
90+
```yaml
91+
kind: operator.local
92+
spec:
93+
image: recommender:v1
94+
volume:
95+
- /Users/<user>/.oci:/root/.oci
96+
- /Users/<user>/recommender/data:/etc/operator/data
97+
- /Users/<user>/recommender/result:/etc/operator/result
98+
type: container
99+
version: v1
100+
```
101+
102+
Run the recommender within a container using the command below:
103+
104+
```bash
105+
ads operator run -f ~/recommender/recommender.yaml --backend-config ~/recommender/backend_operator_local_container_config.yaml
106+
```
107+
108+
## 5. Running recommender in the Data Science job within container runtime
109+
110+
To execute the recommender operator within a Data Science job using container runtime, please follow the steps outlined below:
111+
112+
You can use the following command to build the recommender container. This step can be skipped if you have already done this for running the operator within a local container.
113+
114+
```bash
115+
ads operator build-image -t recommender
116+
```
117+
118+
This will create a new `recommender:v1` image, with `/etc/operator` as the designated working directory within the container.
119+
120+
Publish the `recommender:v1` container to the [Oracle Container Registry](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/home.htm). To become familiar with OCI, read the documentation links posted below.
121+
122+
- [Access Container Registry](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/Concepts/registryoverview.htm#access)
123+
- [Create repositories](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/Tasks/registrycreatingarepository.htm#top)
124+
- [Push images](https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/Tasks/registrypushingimagesusingthedockercli.htm#Pushing_Images_Using_the_Docker_CLI)
125+
126+
To publish `recommender:v1` to OCR, use the command posted below:
127+
128+
```bash
129+
ads operator publish-image recommender:v1 --registry <iad.ocir.io/tenancy/>
130+
```
131+
132+
After the container is published to OCR, it can be used within Data Science jobs service. Check the `backend_job_container_config.yaml` config file. It should contain pre-populated infrastructure and runtime sections. The runtime section should contain an image property, something like `image: iad.ocir.io/<tenancy>/recommender:v1`. More details about supported options can be found in the ADS Jobs documentation - [Run a Container](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/run_container.html).
133+
134+
Adjust the `recommender.yaml` config with proper input/output folders. When the recommender is run in the Data Science job, it will not have access to local folders. Therefore, input data and output folders should be placed in the Object Storage bucket. Open the `recommender.yaml` and adjust the following fields:
135+
136+
```yaml
137+
input_data:
138+
url: oci://bucket@namespace/recommender/input_data/data.csv
139+
output_directory:
140+
url: oci://bucket@namespace/recommender/result/
141+
```
142+
143+
Run the recommender on the Data Science jobs using the command posted below:
144+
145+
```bash
146+
ads operator run -f ~/recommender/recommender.yaml --backend-config ~/recommender/backend_job_container_config.yaml
147+
```
148+
149+
The logs can be monitored using the `ads opctl watch` command.
150+
151+
```bash
152+
ads opctl watch <OCID>
153+
```
154+
155+
## 6. Running recommender in the Data Science job within conda runtime
156+
157+
To execute the recommender operator within a Data Science job using conda runtime, please follow the steps outlined below:
158+
159+
You can use the following command to build the recommender conda environment.
160+
161+
```bash
162+
ads operator build-conda -t recommender
163+
```
164+
165+
This will create a new `recommender_v1` conda environment and place it in the folder specified within `ads opctl configure` command.
166+
167+
Use the command below to Publish the `recommender_v1` conda environment to the Object Storage bucket.
168+
169+
```bash
170+
ads operator publish-conda -t recommender
171+
```
172+
More details about configuring CLI can be found here - [Configuring CLI](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/opctl/configure.html)
173+
174+
175+
After the conda environment is published to Object Storage, it can be used within Data Science jobs service. Check the `backend_job_python_config.yaml` config file. It should contain pre-populated infrastructure and runtime sections. The runtime section should contain a `conda` section.
176+
177+
```yaml
178+
conda:
179+
type: published
180+
uri: oci://bucket@namespace/conda_environments/cpu/recommender/1/recommender_v1
181+
```
182+
183+
More details about supported options can be found in the ADS Jobs documentation - [Run a Python Workload](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/run_python.html).
184+
185+
Adjust the `recommender.yaml` config with proper input/output folders. When the recommender is run in the Data Science job, it will not have access to local folders. Therefore, input data and output folders should be placed in the Object Storage bucket. Open the `recommender.yaml` and adjust the following fields:
186+
187+
```yaml
188+
input_data:
189+
url: oci://bucket@namespace/recommender/input_data/data.csv
190+
output_directory:
191+
url: oci://bucket@namespace/recommender/result/
192+
test_data:
193+
url: oci://bucket@namespace/recommender/input_data/test.csv
194+
```
195+
196+
Run the recommender on the Data Science jobs using the command posted below:
197+
198+
```bash
199+
ads operator run -f ~/recommender/recommender.yaml --backend-config ~/recommender/backend_job_python_config.yaml
200+
```
201+
202+
The logs can be monitored using the `ads opctl watch` command.
203+
204+
```bash
205+
ads opctl watch <OCID>
206+
```
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*--
3+
4+
# Copyright (c) 2023 Oracle and/or its affiliates.
5+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*--
3+
4+
# Copyright (c) 2024 Oracle and/or its affiliates.
5+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
6+
7+
import json
8+
import os
9+
import sys
10+
from typing import Dict, List
11+
12+
import yaml
13+
14+
from ads.opctl import logger
15+
from ads.opctl.operator.common.const import ENV_OPERATOR_ARGS
16+
from ads.opctl.operator.common.utils import _parse_input_args
17+
18+
from .model.recommender_dataset import RecommenderDatasets
19+
from .operator_config import RecommenderOperatorConfig
20+
from .model.factory import RecommenderOperatorModelFactory
21+
22+
def operate(operator_config: RecommenderOperatorConfig) -> None:
23+
"""Runs the recommender operator."""
24+
25+
datasets = RecommenderDatasets(operator_config)
26+
RecommenderOperatorModelFactory.get_model(
27+
operator_config, datasets
28+
).generate_report()
29+
30+
31+
def verify(spec: Dict, **kwargs: Dict) -> bool:
32+
"""Verifies the recommender detection operator config."""
33+
operator = RecommenderOperatorConfig.from_dict(spec)
34+
msg_header = (
35+
f"{'*' * 50} The operator config has been successfully verified {'*' * 50}"
36+
)
37+
print(msg_header)
38+
print(operator.to_yaml())
39+
print("*" * len(msg_header))
40+
41+
42+
def main(raw_args: List[str]):
43+
"""The entry point of the recommender the operator."""
44+
args, _ = _parse_input_args(raw_args)
45+
if not args.file and not args.spec and not os.environ.get(ENV_OPERATOR_ARGS):
46+
logger.info(
47+
"Please specify -f[--file] or -s[--spec] or "
48+
f"pass operator's arguments via {ENV_OPERATOR_ARGS} environment variable."
49+
)
50+
return
51+
52+
logger.info("-" * 100)
53+
logger.info(
54+
f"{'Running' if not args.verify else 'Verifying'} the recommender detection operator."
55+
)
56+
57+
yaml_string = ""
58+
if args.spec or os.environ.get(ENV_OPERATOR_ARGS):
59+
operator_spec_str = args.spec or os.environ.get(ENV_OPERATOR_ARGS)
60+
try:
61+
yaml_string = yaml.safe_dump(json.loads(operator_spec_str))
62+
except json.JSONDecodeError:
63+
yaml_string = yaml.safe_dump(yaml.safe_load(operator_spec_str))
64+
except:
65+
yaml_string = operator_spec_str
66+
67+
operator_config = RecommenderOperatorConfig.from_yaml(
68+
uri=args.file,
69+
yaml_string=yaml_string,
70+
)
71+
72+
logger.info(operator_config.to_yaml())
73+
74+
# run operator
75+
if args.verify:
76+
verify(operator_config)
77+
else:
78+
operate(operator_config)
79+
80+
81+
if __name__ == "__main__":
82+
main(sys.argv[1:])
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/usr/bin/env python
2+
# -*- coding: utf-8 -*--
3+
4+
# Copyright (c) 2023 Oracle and/or its affiliates.
5+
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
6+
7+
from typing import Dict
8+
9+
from ads.opctl.operator.common.operator_yaml_generator import YamlGenerator
10+
from ads.opctl.operator.common.utils import _load_yaml_from_uri
11+
12+
13+
def init(**kwargs: Dict) -> str:
14+
"""
15+
Generates operator config by the schema.
16+
17+
Properties
18+
----------
19+
kwargs: (Dict, optional).
20+
Additional key value arguments.
21+
22+
- type: str
23+
The type of the operator.
24+
25+
Returns
26+
-------
27+
str
28+
The YAML specification generated based on the schema.
29+
"""
30+
31+
return YamlGenerator(
32+
schema=_load_yaml_from_uri(__file__.replace("cmd.py", "schema.yaml"))
33+
).generate_example_dict(values={"type": kwargs.get("type")})

0 commit comments

Comments
 (0)