Update doc for feature store (#361)

Author: KshitizLohia

ads/feature_store/docs/source/figures/feature_store_overview.png

@@ -29,3 +29,8 @@ Notebook Examples
       - `Schema enforcement and schema evolution <https://objectstorage.us-ashburn-1.oraclecloud.com/p/hh2NOgFJbVSg4amcLM3G3hkTuHyBD-8aE_iCsuZKEvIav1Wlld-3zfCawG4ycQGN/n/ociodscdev/b/oci-feature-store/o/beta/notebook/feature_store_flights_schema_evolution.ipynb>`__
       - | 1. Schema evolution is a feature that allows users to easily change a table's current schema to accommodate data that is changing over time.
         | 2. Schema enforcement, also known as schema validation, is a safeguard in Delta Lake that ensures data quality by rejecting writes to a table that do not match the table's schema.
+
+    * - `Embeddings in Feature Store <https://objectstorage.us-ashburn-1.oraclecloud.com/p/hh2NOgFJbVSg4amcLM3G3hkTuHyBD-8aE_iCsuZKEvIav1Wlld-3zfCawG4ycQGN/n/ociodscdev/b/oci-feature-store/o/beta/notebook/feature_store_embeddings.html>`__
+      - `Embeddings in Feature Store <https://objectstorage.us-ashburn-1.oraclecloud.com/p/hh2NOgFJbVSg4amcLM3G3hkTuHyBD-8aE_iCsuZKEvIav1Wlld-3zfCawG4ycQGN/n/ociodscdev/b/oci-feature-store/o/beta/notebook/feature_store_embeddings.ipynb>`__
+      - | 1. One of the primary functions of an embedding feature store is to store pre-trained word embeddings, such as Word2Vec, GloVe, FastText, or BERT embeddings. These embeddings are learned from massive text corpora and contain information about word semantics, which can be valuable for various NLP tasks like text classification, named entity recognition, sentiment analysis, and more.
+        | 2. Embedding feature stores are optimized for fast and efficient retrieval of embeddings. This is crucial because embeddings can be high-dimensional and computationally expensive to calculate. By storing them in a dedicated store, you can avoid the need to recalculate embeddings for the same data repeatedly.

ads/feature_store/docs/source/notebook.rst

@@ -3,7 +3,7 @@ Overview
 
 Managing many datasets, data-sources and transformations for machine learning is complex and costly. Poorly cleaned data, data issues, bugs in transformations, data drift and training serving skew all leads to increased model development time and worse model performance. Here, feature store is well positioned to solve many of the problems since it provides a centralised way to transform and access data for training and serving time and helps defines a standardised pipeline for ingestion of data and querying of data.
 
-.. image:: figures/overview.png
+.. image:: figures/feature_store_overview.png
 
 Oracle feature store is a stack based solution that is deployed in the customer enclave using OCI resource manager. Customer can stand up the service with infrastructure in their own tenancy. The service consists of API which are deployed in customer tenancy using resource manager.
 

ads/feature_store/docs/source/overview.rst

@@ -28,7 +28,7 @@ Feature Store users need to provide the following access permissions in order to
 
 ..  code-block:: shell
 
-    define tenancy service_tenancy as ocid1.tenancy.oc1..aaaaaaaahzy3x4boh7ipxyft2rowu2xeglvanlfewudbnueugsieyuojkldq
+    define tenancy service_tenancy as ocid1.tenancy.oc1..aaaaaaaa462hfhplpx652b32ix62xrdijppq2c7okwcqjlgrbknhgtj2kofa
     endorse group <feature store user group> to read repos in tenancy service_tenancy
     allow group <feature store user group> to manage orm-stacks in compartment <compartmentName>
     allow group <feature store user group> to manage orm-jobs in compartment <compartmentName>

ads/feature_store/docs/source/terraform.rst

@@ -16,40 +16,23 @@
     )
 
 
-class AbsFeatureStat:
-    class ValidationFailedException(Exception):
-        def __init__(self):
-            pass
-
+class AbsFeatureValue:
     def __init__(self):
         self.__validate__()
 
     @abstractmethod
     def __validate__(self):
         pass
 
-    @abstractmethod
-    def add_to_figure(self, fig: Figure, xaxis: int, yaxis: int):
-        pass
-
     @classmethod
     @abstractmethod
     def __from_json__(cls, json_dict: dict):
         pass
 
-    @staticmethod
-    def get_x_y_str_axes(xaxis: int, yaxis: int) -> ():
-        return (
-            ("xaxis" + str(xaxis + 1)),
-            ("yaxis" + str(yaxis + 1)),
-            ("x" + str(xaxis + 1)),
-            ("y" + str(yaxis + 1)),
-        )
-
     @classmethod
     def from_json(
         cls, json_dict: dict, ignore_errors: bool = False
-    ) -> Union["AbsFeatureStat", None]:
+    ) -> Union["AbsFeatureValue", None]:
         try:
             return cls.__from_json__(json_dict=json_dict)
         except Exception as e:

ads/feature_store/statistics/charts/abstract_feature_stat.py renamed to ads/feature_store/statistics/abs_feature_value.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+# -*- coding: utf-8; -*-
+# Copyright (c) 2023 Oracle and/or its affiliates.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
+import abc
+
+from abc import abstractmethod
+from ads.feature_store.statistics.abs_feature_value import AbsFeatureValue
+from ads.common.decorator.runtime_dependency import OptionalDependency
+
+try:
+    from plotly.graph_objs import Figure
+except ModuleNotFoundError:
+    raise ModuleNotFoundError(
+        f"The `plotly` module was not found. Please run `pip install "
+        f"{OptionalDependency.FEATURE_STORE}`."
+    )
+
+
+class AbsFeaturePlot(abc.ABC, AbsFeatureValue):
+    @abstractmethod
+    def add_to_figure(self, fig: Figure, xaxis: int, yaxis: int):
+        pass
+
+    @classmethod
+    @abstractmethod
+    def __from_json__(cls, json_dict: dict):
+        pass
+
+    @staticmethod
+    def get_x_y_str_axes(xaxis: int, yaxis: int) -> ():
+        return (
+            ("xaxis" + str(xaxis + 1)),
+            ("yaxis" + str(yaxis + 1)),
+            ("x" + str(xaxis + 1)),
+            ("y" + str(yaxis + 1)),
+        )

ads/feature_store/statistics/charts/abstract_feature_plot.py

@@ -2,10 +2,12 @@
 # -*- coding: utf-8; -*-
 # Copyright (c) 2023 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
-from typing import List
+from typing import List, Union
+
+from ads.feature_store.statistics.abs_feature_value import AbsFeatureValue
 
 from ads.common.decorator.runtime_dependency import OptionalDependency
-from ads.feature_store.statistics.charts.abstract_feature_stat import AbsFeatureStat
+from ads.feature_store.statistics.charts.abstract_feature_plot import AbsFeaturePlot
 from ads.feature_store.statistics.charts.frequency_distribution import (
     FrequencyDistribution,
 )
@@ -20,7 +22,7 @@
     )
 
 
-class BoxPlot(AbsFeatureStat):
+class BoxPlot(AbsFeaturePlot):
     CONST_MIN = "Min"
     CONST_MAX = "Max"
     CONST_QUARTILES = "Quartiles"
@@ -30,7 +32,7 @@ class BoxPlot(AbsFeatureStat):
     CONST_IQR = "IQR"
     CONST_BOX_POINTS = "box_points"
 
-    class Quartiles:
+    class Quartiles(AbsFeatureValue):
         CONST_Q1 = "q1"
         CONST_Q2 = "q2"
         CONST_Q3 = "q3"
@@ -39,15 +41,20 @@ def __init__(self, q1: float, q2: float, q3: float):
             self.q1 = q1
             self.q2 = q2
             self.q3 = q3
+            super().__init__()
 
         @classmethod
-        def from_json(cls, json_dict: dict) -> "BoxPlot.Quartiles":
+        def __from_json__(cls, json_dict: dict) -> "BoxPlot.Quartiles":
             return cls(
                 json_dict.get(cls.CONST_Q1),
                 json_dict.get(cls.CONST_Q2),
                 json_dict.get(cls.CONST_Q3),
             )
 
+        def __validate__(self):
+            assert type(self.q1) == type(self.q2) == type(self.q3) == int or float
+            assert self.q3 >= self.q2 >= self.q1
+
     def __init__(
         self,
         mean: float,
@@ -67,14 +74,14 @@ def __init__(
         super().__init__()
 
     def __validate__(self):
-        if (
-            self.q1 is None
-            or self.q3 is None
-            or self.iqr is None
-            or type(self.box_points) is not list
-            or len(self.box_points) == 0
-        ):
-            return self.ValidationFailedException()
+        assert self.q1 is not None
+        assert self.q3 is not None
+        assert self.iqr is not None
+        assert self.q3 is not None
+        assert self.median is not None
+        assert self.mean is not None
+        assert type(self.box_points) is list
+        assert len(self.box_points) > 0
 
     def add_to_figure(self, fig: Figure, xaxis: int, yaxis: int):
         xaxis_str, yaxis_str, x_str, y_str = self.get_x_y_str_axes(xaxis, yaxis)

ads/feature_store/statistics/charts/box_plot.py

@@ -5,7 +5,7 @@
 
 from typing import List
 from ads.common.decorator.runtime_dependency import OptionalDependency
-from ads.feature_store.statistics.charts.abstract_feature_stat import AbsFeatureStat
+from ads.feature_store.statistics.charts.abstract_feature_plot import AbsFeaturePlot
 
 try:
     from plotly.graph_objs import Figure
@@ -16,24 +16,21 @@
     )
 
 
-class FrequencyDistribution(AbsFeatureStat):
+class FrequencyDistribution(AbsFeaturePlot):
     CONST_FREQUENCY = "frequency"
     CONST_BINS = "bins"
     CONST_FREQUENCY_DISTRIBUTION_TITLE = "Frequency Distribution"
 
-    def __validate__(self):
-        if not (
-            type(self.frequency) == list
-            and type(self.bins) == list
-            and 0 < len(self.frequency) == len(self.bins) > 0
-        ):
-            raise self.ValidationFailedException()
-
     def __init__(self, frequency: List, bins: List):
         self.frequency = frequency
         self.bins = bins
         super().__init__()
 
+    def __validate__(self):
+        assert type(self.frequency) == list
+        assert type(self.bins) == list
+        assert 0 < len(self.frequency) == len(self.bins) > 0
+
     @classmethod
     def __from_json__(cls, json_dict: dict) -> "FrequencyDistribution":
         return FrequencyDistribution(

ads/feature_store/statistics/charts/frequency_distribution.py

@@ -5,7 +5,7 @@
 from typing import List
 
 from ads.common.decorator.runtime_dependency import OptionalDependency
-from ads.feature_store.statistics.charts.abstract_feature_stat import AbsFeatureStat
+from ads.feature_store.statistics.charts.abstract_feature_plot import AbsFeaturePlot
 
 try:
     from plotly.graph_objs import Figure
@@ -16,15 +16,7 @@
     )
 
 
-class ProbabilityDistribution(AbsFeatureStat):
-    def __validate__(self):
-        if not (
-            type(self.density) == list
-            and type(self.bins) == list
-            and 0 < len(self.density) == len(self.bins) > 0
-        ):
-            raise self.ValidationFailedException()
-
+class ProbabilityDistribution(AbsFeaturePlot):
     CONST_DENSITY = "density"
     CONST_BINS = "bins"
     CONST_PROBABILITY_DISTRIBUTION_TITLE = "Probability Distribution"
@@ -34,6 +26,11 @@ def __init__(self, density: List, bins: List):
         self.bins = bins
         super().__init__()
 
+    def __validate__(self):
+        assert type(self.density) == list
+        assert type(self.bins) == list
+        assert 0 < len(self.density) == len(self.bins) > 0
+
     @classmethod
     def __from_json__(cls, json_dict: dict) -> "ProbabilityDistribution":
         return cls(

ads/feature_store/statistics/charts/probability_distribution.py

@@ -3,9 +3,12 @@
 # Copyright (c) 2023 Oracle and/or its affiliates.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/
 from typing import List
+
+from ads.feature_store.statistics.abs_feature_value import AbsFeatureValue
+
 from ads.common.decorator.runtime_dependency import OptionalDependency
 
-from ads.feature_store.statistics.charts.abstract_feature_stat import AbsFeatureStat
+from ads.feature_store.statistics.charts.abstract_feature_plot import AbsFeaturePlot
 
 try:
     from plotly.graph_objs import Figure
@@ -16,15 +19,11 @@
     )
 
 
-class TopKFrequentElements(AbsFeatureStat):
-    def __validate__(self):
-        if not (type(self.elements) == list and len(self.elements) > 0):
-            raise self.ValidationFailedException
-
+class TopKFrequentElements(AbsFeaturePlot):
     CONST_VALUE = "value"
     CONST_TOP_K_FREQUENT_TITLE = "Top K Frequent Elements"
 
-    class TopKFrequentElement:
+    class TopKFrequentElement(AbsFeatureValue):
         CONST_VALUE = "value"
         CONST_ESTIMATE = "estimate"
         CONST_LOWER_BOUND = "lower_bound"
@@ -37,9 +36,14 @@ def __init__(
             self.estimate = estimate
             self.lower_bound = lower_bound
             self.upper_bound = upper_bound
+            super().__init__()
+
+        def __validate__(self):
+            assert type(self.value) == str and len(self.value) > 0
+            assert type(self.estimate) == int and self.estimate >= 0
 
         @classmethod
-        def from_json(
+        def __from_json__(
             cls, json_dict: dict
         ) -> "TopKFrequentElements.TopKFrequentElement":
             return cls(
@@ -53,6 +57,12 @@ def __init__(self, elements: List[TopKFrequentElement]):
         self.elements = elements
         super().__init__()
 
+    def __validate__(self):
+        assert type(self.elements) == list
+        assert len(self.elements) > 0
+        for element in self.elements:
+            assert element is not None
+
     @classmethod
     def __from_json__(cls, json_dict: dict) -> "TopKFrequentElements":
         elements = json_dict.get(cls.CONST_VALUE)