Merge branch 'main' into ODSC-65658/llamaindex_docs

Author: mrDzurb

.gitignore

@@ -86,7 +86,8 @@ celerybeat-schedule
 *.sage.py
 
 # dotenv
-.env
+.env*
+run_ads.sh
 
 # virtualenv
 .venv

README-development.md

@@ -1,15 +1,41 @@
+<!-- TOC -->
 # Summary
 
 The Oracle Accelerated Data Science (ADS) SDK used by data scientists and analysts for
 data exploration and experimental machine learning to democratize machine learning and
-analytics by providing easy-to-use, performant, and user friendly tools that
+analytics by providing easy-to-use, 
+performant, and user friendly tools that
 brings together the best of data science practices.
 
 The ADS SDK helps you connect to different data sources, perform exploratory data analysis,
 data visualization, feature engineering, model training, model evaluation, and
 model interpretation. ADS also allows you to connect to the model catalog to save and load
 models to and from the catalog.
 
+- [Summary](#summary)
+  - [Documentation](#documentation)
+  - [Get Support](#get-support)
+  - [Getting started](#getting-started)
+    - [Step 1: Create a conda environment](#step-1-create-a-conda-environment)
+    - [Step 2: Activate your environment](#step-2-activate-your-environment)
+    - [Step 3: Clone ADS and install dependencies](#step-3-clone-ads-and-install-dependencies)
+    - [Step 4: Setup configuration files](#step-4-setup-configuration-files)
+    - [Step 5: Versioning and generation the wheel](#step-5-versioning-and-generation-the-wheel)
+  - [Running tests](#running-tests)
+    - [Running default setup tests](#running-default-setup-tests)
+    - [Running all unit tests](#running-all-unit-tests)
+    - [Running integration tests](#running-integration-tests)
+    - [Running opctl integration tests](#running-opctl-integration-tests)
+  - [Local Setup of AQUA API JupyterLab Server](#local-setup-of-aqua-api-jupyterlab-server)
+    - [Step 1: Requirements](#step-1-requirements)
+    - [Step 2: Create local .env files](#step-2-create-local-env-files)
+    - [Step 3: Add the run\_ads.sh script in the ADS Repository](#step-3-add-the-run_adssh-script-in-the-ads-repository)
+    - [Step 4: Run the JupyterLab Server](#step-4-run-the-jupyterlab-server)
+    - [Step 5: Run the unit tests for the AQUA API](#step-5-run-the-unit-tests-for-the-aqua-api)
+  - [Security](#security)
+  - [License](#license)
+
+
 ## Documentation
 
  - [ads-documentation](https://docs.oracle.com/en-us/iaas/tools/ads-sdk/latest/index.html)
@@ -137,6 +163,100 @@ To build development container, see the [Build Development Container Image](http
   python3 -m pytest tests/integration/opctl
 ```
 
+## Local Setup of AQUA API JupyterLab Server
+These are the steps to run the AQUA (AI Quick Actions) API Server for development and testing purposes. The source code for the AQUA API Server is [here](https://github.com/oracle/accelerated-data-science/tree/21ba00b95aef8581991fee6c7d558e2f2b1680ac/ads/aqua) within this repository.
+
+### Step 1: Requirements
++ Complete the [Getting Started](#getting-started) Section above, create a conda environment with python >3.9 or 3.10
++ install any Rest API Client in your IDE (Thunder Client on [vscode](https://marketplace.visualstudio.com/items?itemName=rangav.vscode-thunder-client) or Postman) 
++ Activate the conda environment from the Getting Started Section and run
+
+```
+pip install -r test-requirements.txt
+```
+
+### Step 2: Create local .env files 
+Running the local JuypterLab server requires setting OCI authentication, proxy, and OCI namespace parameters. Adapt this .env file with your specific OCI profile and OCIDs to set these variables.
+
+```
+CONDA_BUCKET_NS="your_conda_bucket"
+http_proxy=""
+https_proxy=""
+HTTP_PROXY=""
+HTTPS_PROXY=""
+OCI_ODSC_SERVICE_ENDPOINT="your_service_endpoint"
+AQUA_SERVICE_MODELS_BUCKET="service-managed-models"
+AQUA_TELEMETRY_BUCKET_NS="" 
+PROJECT_COMPARTMENT_OCID="ocid1.compartment.oc1.<your_ocid>" 
+OCI_CONFIG_PROFILE="your_oci_profile_name"
+OCI_IAM_TYPE="security_token" # no modification needed if using token-based auth
+TENANCY_OCID="ocid1.tenancy.oc1.<your_ocid>"
+AQUA_JOB_SUBNET_ID="ocid1.subnet.oc1.<your_ocid>"
+ODSC_MODEL_COMPARTMENT_OCID="ocid1.compartment.oc1.<your_ocid>" 
+PROJECT_OCID="ocid1.datascienceproject.oc1.<your_ocid>"
+```
+
+### Step 3: Add the run_ads.sh script in the ADS Repository 
++ add the shell script below and .env file from step 2 to your local directory of the cloned ADS Repository
++ Run ```chmox +x run_ads.sh``` after you create this script.
+```
+#!/bin/bash
+
+#### Check if a CLI command is provided
+if [ "$#" -lt 1 ]; then
+  echo "Usage: $0 <cli command>"
+  exit 1
+fi
+
+#### Load environment variables from .env file
+if [ -f .env ]; then
+  export $(grep -v '^#' .env.int | xargs)
+else
+  echo "Error: .env.int file not found!"
+  exit 1
+fi
+
+# Execute the CLI command
+"$@"
+```
+
+### Step 4: Run the JupyterLab Server 
+We can start the JupyterLab server using the following command
+
+``` 
+./run_ads.sh jupyter lab --no-browser --ServerApp.disable_check_xsrf=True
+```
++ run ```pkill jupyter-lab``` to kill the JupyterLab server and re-run server to reflect changes made locally to the AQUA API
++ to test if server is running via CLI, run this in terminal
+
+```
+./run_ads.sh ads aqua model list
+```
+
+To make calls to the API, use the link http://localhost:8888/aqua/insert_handler_here with a REST API Client like Thunder Client/ Postman.
+
+Examples of handlers
+```
+GET http://localhost:8888/aqua/model # calling the model_handler.py
+
+GET http://localhost:8888/aqua/deployments # calling the deployment_handler.py
+```
+Handlers can be found [here](https://github.com/oracle/accelerated-data-science/tree/21ba00b95aef8581991fee6c7d558e2f2b1680ac/ads/aqua/extension).
+
+### Step 5: Run the unit tests for the AQUA API
+All the unit tests can be found [here](https://github.com/oracle/accelerated-data-science/tree/main/tests/unitary/with_extras/aqua). 
+The following commands detail how the unit tests can be run.
+```
+# Run all tests in AQUA project
+python -m pytest -q tests/unitary/with_extras/aqua/test_deployment.py
+
+# Run all tests specific to a module within in AQUA project (ex. test_deployment.py, test_model.py, etc.)
+python -m pytest -q tests/unitary/with_extras/aqua/test_deployment.py
+
+# Run specific test method within the module (replace test_get_deployment_default_params with targeted test method)
+python -m pytest tests/unitary/with_extras/aqua/test_deployment.py -k "test_get_deployment_default_params"
+```
+
 ## Security
 
 Consult the [security guide](./SECURITY.md) for our responsible security

ads/aqua/common/enums.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# Copyright (c) 2024 Oracle and/or its affiliates.
+# Copyright (c) 2024, 2025 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
 
 """
@@ -8,23 +8,23 @@
 This module contains the set of enums used in AQUA.
 """
 
-from ads.common.extended_enum import ExtendedEnumMeta
+from ads.common.extended_enum import ExtendedEnum
 
 
-class DataScienceResource(str, metaclass=ExtendedEnumMeta):
+class DataScienceResource(ExtendedEnum):
     MODEL_DEPLOYMENT = "datasciencemodeldeployment"
     MODEL = "datasciencemodel"
 
 
-class Resource(str, metaclass=ExtendedEnumMeta):
+class Resource(ExtendedEnum):
     JOB = "jobs"
     JOBRUN = "jobruns"
     MODEL = "models"
     MODEL_DEPLOYMENT = "modeldeployments"
     MODEL_VERSION_SET = "model-version-sets"
 
 
-class Tags(str, metaclass=ExtendedEnumMeta):
+class Tags(ExtendedEnum):
     TASK = "task"
     LICENSE = "license"
     ORGANIZATION = "organization"
@@ -42,41 +42,41 @@ class Tags(str, metaclass=ExtendedEnumMeta):
     MODEL_ARTIFACT_FILE = "model_file"
 
 
-class InferenceContainerType(str, metaclass=ExtendedEnumMeta):
+class InferenceContainerType(ExtendedEnum):
     CONTAINER_TYPE_VLLM = "vllm"
     CONTAINER_TYPE_TGI = "tgi"
     CONTAINER_TYPE_LLAMA_CPP = "llama-cpp"
 
 
-class InferenceContainerTypeFamily(str, metaclass=ExtendedEnumMeta):
+class InferenceContainerTypeFamily(ExtendedEnum):
     AQUA_VLLM_CONTAINER_FAMILY = "odsc-vllm-serving"
     AQUA_TGI_CONTAINER_FAMILY = "odsc-tgi-serving"
     AQUA_LLAMA_CPP_CONTAINER_FAMILY = "odsc-llama-cpp-serving"
 
 
-class CustomInferenceContainerTypeFamily(str, metaclass=ExtendedEnumMeta):
+class CustomInferenceContainerTypeFamily(ExtendedEnum):
     AQUA_TEI_CONTAINER_FAMILY = "odsc-tei-serving"
 
 
-class InferenceContainerParamType(str, metaclass=ExtendedEnumMeta):
+class InferenceContainerParamType(ExtendedEnum):
     PARAM_TYPE_VLLM = "VLLM_PARAMS"
     PARAM_TYPE_TGI = "TGI_PARAMS"
     PARAM_TYPE_LLAMA_CPP = "LLAMA_CPP_PARAMS"
 
 
-class EvaluationContainerTypeFamily(str, metaclass=ExtendedEnumMeta):
+class EvaluationContainerTypeFamily(ExtendedEnum):
     AQUA_EVALUATION_CONTAINER_FAMILY = "odsc-llm-evaluate"
 
 
-class FineTuningContainerTypeFamily(str, metaclass=ExtendedEnumMeta):
+class FineTuningContainerTypeFamily(ExtendedEnum):
     AQUA_FINETUNING_CONTAINER_FAMILY = "odsc-llm-fine-tuning"
 
 
-class HuggingFaceTags(str, metaclass=ExtendedEnumMeta):
+class HuggingFaceTags(ExtendedEnum):
     TEXT_GENERATION_INFERENCE = "text-generation-inference"
 
 
-class RqsAdditionalDetails(str, metaclass=ExtendedEnumMeta):
+class RqsAdditionalDetails(ExtendedEnum):
     METADATA = "metadata"
     CREATED_BY = "createdBy"
     DESCRIPTION = "description"
@@ -86,7 +86,7 @@ class RqsAdditionalDetails(str, metaclass=ExtendedEnumMeta):
     VERSION_LABEL = "versionLabel"
 
 
-class TextEmbeddingInferenceContainerParams(str, metaclass=ExtendedEnumMeta):
+class TextEmbeddingInferenceContainerParams(ExtendedEnum):
     """Contains a subset of params that are required for enabling model deployment in OCI Data Science. More options
     are available at https://huggingface.co/docs/text-embeddings-inference/en/cli_arguments"""
 

ads/aqua/common/errors.py

@@ -1,6 +1,5 @@
 #!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright (c) 2024 Oracle and/or its affiliates.
+# Copyright (c) 2024, 2025 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
 
 """
@@ -10,10 +9,10 @@
 This module contains the set of Aqua exceptions.
 """
 
-from ads.common.extended_enum import ExtendedEnumMeta
+from ads.common.extended_enum import ExtendedEnum
 
 
-class ExitCode(str, metaclass=ExtendedEnumMeta):
+class ExitCode(ExtendedEnum):
     SUCCESS = 0
     COMMON_ERROR = 1
     INVALID_CONFIG = 10

ads/aqua/common/utils.py

@@ -65,7 +65,7 @@
 from ads.aqua.data import AquaResourceIdentifier
 from ads.common.auth import AuthState, default_signer
 from ads.common.decorator.threaded import threaded
-from ads.common.extended_enum import ExtendedEnumMeta
+from ads.common.extended_enum import ExtendedEnum
 from ads.common.object_storage_details import ObjectStorageDetails
 from ads.common.oci_resource import SEARCH_TYPE, OCIResource
 from ads.common.utils import copy_file, get_console_link, upload_to_os
@@ -80,7 +80,7 @@
 logger = logging.getLogger("ads.aqua")
 
 
-class LifecycleStatus(str, metaclass=ExtendedEnumMeta):
+class LifecycleStatus(ExtendedEnum):
     UNKNOWN = ""
 
     @property

ads/aqua/evaluation/constants.py

@@ -1,6 +1,5 @@
 #!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright (c) 2024 Oracle and/or its affiliates.
+# Copyright (c) 2024, 2025 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
 
 """
@@ -9,17 +8,18 @@
 
 This module contains constants/enums used in Aqua Evaluation.
 """
+
 from oci.data_science.models import JobRun
 
-from ads.common.extended_enum import ExtendedEnumMeta
+from ads.common.extended_enum import ExtendedEnum
 
 EVAL_TERMINATION_STATE = [
     JobRun.LIFECYCLE_STATE_SUCCEEDED,
     JobRun.LIFECYCLE_STATE_FAILED,
 ]
 
 
-class EvaluationCustomMetadata(str, metaclass=ExtendedEnumMeta):
+class EvaluationCustomMetadata(ExtendedEnum):
     EVALUATION_SOURCE = "evaluation_source"
     EVALUATION_JOB_ID = "evaluation_job_id"
     EVALUATION_JOB_RUN_ID = "evaluation_job_run_id"
@@ -28,11 +28,11 @@ class EvaluationCustomMetadata(str, metaclass=ExtendedEnumMeta):
     EVALUATION_ERROR = "aqua_evaluate_error"
 
 
-class EvaluationConfig(str, metaclass=ExtendedEnumMeta):
+class EvaluationConfig(ExtendedEnum):
     PARAMS = "model_params"
 
 
-class EvaluationReportJson(str, metaclass=ExtendedEnumMeta):
+class EvaluationReportJson(ExtendedEnum):
     """Contains evaluation report.json fields name."""
 
     METRIC_SUMMARY_RESULT = "metric_summary_result"
@@ -43,7 +43,7 @@ class EvaluationReportJson(str, metaclass=ExtendedEnumMeta):
     DATASET = "dataset"
 
 
-class EvaluationMetricResult(str, metaclass=ExtendedEnumMeta):
+class EvaluationMetricResult(ExtendedEnum):
     """Contains metric result's fields name in report.json."""
 
     SHORT_NAME = "key"

ads/aqua/evaluation/errors.py

@@ -1,6 +1,5 @@
 #!/usr/bin/env python
-# -*- coding: utf-8 -*-
-# Copyright (c) 2024 Oracle and/or its affiliates.
+# Copyright (c) 2024, 2025 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
 """
 aqua.evaluation.errors
@@ -9,10 +8,10 @@
 This module contains errors in Aqua Evaluation.
 """
 
-from ads.common.extended_enum import ExtendedEnumMeta
+from ads.common.extended_enum import ExtendedEnum
 
 
-class EvaluationJobExitCode(str, metaclass=ExtendedEnumMeta):
+class EvaluationJobExitCode(ExtendedEnum):
     SUCCESS = 0
     COMMON_ERROR = 1
 

ads/aqua/extension/models/ws_models.py

@@ -1,20 +1,19 @@
 #!/usr/bin/env python
-# -*- coding: utf-8 -*--
 
-# Copyright (c) 2024 Oracle and/or its affiliates.
+# Copyright (c) 2024, 2025 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
 
 from dataclasses import dataclass
 from typing import List, Optional
 
-from ads.aqua.evaluation.entities import AquaEvaluationSummary, AquaEvaluationDetail
-from ads.aqua.model.entities import AquaModelSummary, AquaModel
+from ads.aqua.evaluation.entities import AquaEvaluationDetail, AquaEvaluationSummary
+from ads.aqua.model.entities import AquaModel, AquaModelSummary
 from ads.aqua.modeldeployment.entities import AquaDeployment, AquaDeploymentDetail
-from ads.common.extended_enum import ExtendedEnumMeta
+from ads.common.extended_enum import ExtendedEnum
 from ads.common.serializer import DataClassSerializable
 
 
-class RequestResponseType(str, metaclass=ExtendedEnumMeta):
+class RequestResponseType(ExtendedEnum):
     ListEvaluations = "ListEvaluations"
     EvaluationDetails = "EvaluationDetails"
     ListDeployments = "ListDeployments"