Merge branch 'issue_86' into issue_86_snapshot_silver

Author: ravi-databricks

.github/workflows/gh_pages.yml

@@ -9,21 +9,21 @@ defaults:
     working-directory: ./docs
 jobs:
   deploy:
-    runs-on: ubuntu-20.04
-    concurrency:
-      group: ${{ github.workflow }}-${{ github.ref }}
+    runs-on:
+      group: databrickslabs-protected-runner-group
+      labels: linux-ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
         with:
           submodules: true  # Fetch Hugo themes (true OR recursive)
           fetch-depth: 0    # Fetch all history for .GitInfo and .Lastmod        
 
 
       - name: Setup Hugo
-        uses: peaceiris/actions-hugo@v2
+        uses: peaceiris/actions-hugo@v3
         with:
-          hugo-version: '0.87.0'
+          hugo-version: '0.143.0'
           extended: true
 
       - name: Build
@@ -35,4 +35,4 @@ jobs:
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_dir: ./docs/public
-          publish_branch: public_docs_v1
+          publish_branch: public_docs_v1

.github/workflows/onpush.yml

@@ -33,10 +33,30 @@ jobs:
 
       - name: Install coverage
         run: pip install coverage
-      
+
+      - name: Install psutil
+        run: pip install psutil      
+
       - name: Lint
-        run: flake8
+        run: flake8        
+
+      - name: set spark local
+        run: export SPARK_LOCAL_IP=127.0.0.1
+
+      - name: set spark executor memory
+        run: export SPARK_EXECUTOR_MEMORY=8g  
+      
+      - name: set spark driver memory
+        run: export SPARK_DRIVER_MEMORY=8g
+
+      - name: set javaopts
+        run: export JAVA_OPTS="-Xmx10g -XX:+UseG1GC"
 
+      - name: Print System Information
+        run: | 
+               python -c "import psutil; import os;
+               print(f'Physical Memory: {psutil.virtual_memory().total / 1e9:.2f} GB'); print(f'CPU Cores: {os.cpu_count()}')"
+ 
       - name: Run Unit Tests
         run: python -m coverage run
 
@@ -51,4 +71,4 @@ jobs:
           name: codecov-umbrella
           path_to_write_report: ./coverage/codecov_report.txt
           verbose: true
- 
+ 

.gitignore

@@ -155,8 +155,7 @@ deployment-merged.yaml
 demo/conf/onboarding.json
 integration_tests/conf/onboarding*.json
 demo/conf/onboarding*.json
-databricks.yaml
 integration_test_output*.csv
 databricks.yml
+oboarding_job_details.json
 
-.databricks

.gitmodules

@@ -4,3 +4,6 @@
 [submodule "docs/themes/hugo-theme-learn"]
 	path = docs/themes/hugo-theme-learn
 	url = https://github.com/matcornic/hugo-theme-learn.git
+[submodule "docs/themes/hugo-theme-relearn"]
+	path = docs/themes/hugo-theme-relearn
+	url = https://github.com/McShelby/hugo-theme-relearn.git

CHANGELOG.md

@@ -1,18 +1,22 @@
 # Changelog
 ## [v.0.0.9] 
-- Added dlt append_flow api support: [PR](https://github.com/databrickslabs/dlt-meta/pull/58)
+- Added  apply_changes_from_snapshot api support in bronze layer: [PR](https://github.com/databrickslabs/dlt-meta/pull/124)
 - Added dlt append_flow api support for silver layer: [PR](https://github.com/databrickslabs/dlt-meta/pull/63)
 - Added support for file metadata columns for autoloader: [PR](https://github.com/databrickslabs/dlt-meta/pull/56)
 - Added support for Bring your own custom transformation: [Issue](https://github.com/databrickslabs/dlt-meta/issues/68)
 - Added support to Unify PyPI releases with GitHub OIDC: [PR](https://github.com/databrickslabs/dlt-meta/pull/62)
 - Added demo for append_flow and file_metadata options: [PR](https://github.com/databrickslabs/dlt-meta/issues/74)
 - Added Demo for silver fanout architecture: [PR](https://github.com/databrickslabs/dlt-meta/pull/83)
-- Added documentation in docs site for new features: [PR](https://github.com/databrickslabs/dlt-meta/pull/64)
+- Added  hugo-theme-relearn themee: [PR](https://github.com/databrickslabs/dlt-meta/pull/132)
 - Added unit tests to showcase silver layer fanout examples: [PR](https://github.com/databrickslabs/dlt-meta/pull/67)
+- Added liquid cluster support: [PR](https://github.com/databrickslabs/dlt-meta/pull/136)
+- Added support for UC Volume + Serverless support for CLI, Integration tests and Demos: [PR](https://github.com/databrickslabs/dlt-meta/pull/105)
+- Added Chaining bronze/silver pipelines into single DLT: [PR](https://github.com/databrickslabs/dlt-meta/pull/130)
 - Fixed issue for No such file or directory: '/demo' :[PR](https://github.com/databrickslabs/dlt-meta/issues/59)
 - Fixed issue DLT-META CLI onboard command issue for Azure: databricks.sdk.errors.platform.ResourceAlreadyExists :[PR](https://github.com/databrickslabs/dlt-meta/issues/51)
 - Fixed issue Changed dbfs.create to mkdirs for CLI: [PR](https://github.com/databrickslabs/dlt-meta/pull/53)
 - Fixed issue DLT-META CLI should use pypi lib instead of whl : [PR](https://github.com/databrickslabs/dlt-meta/pull/79)
+- Fixed issue Onboarding with multiple partition columns errors out: [PR](https://github.com/databrickslabs/dlt-meta/pull/134)
 
 ## [v.0.0.7] 
 - Added dlt-meta cli documentation and readme with browser support: [PR](https://github.com/databrickslabs/dlt-meta/pull/45)

README.md

@@ -68,6 +68,21 @@ In practice, a single generic DLT pipeline reads the Dataflowspec and uses it to
 
 ![DLT-META Stages](./docs/static/images/dlt-meta_stages.png)
 
+## DLT-META DLT Features support
+| Features  | DLT-META Support |
+| ------------- | ------------- |
+| Input data sources  | Autoloader, Delta, Eventhub, Kafka, snapshot  |
+| Medallion architecture layers | Bronze, Silver  |
+| Custom transformations | Bronze, Silver layer accepts custom functions|
+| Data Quality Expecations Support | Bronze, Silver layer |
+| Quarantine table support | Bronze layer |
+| [apply_changes](https://docs.databricks.com/en/delta-live-tables/python-ref.html#cdc) API support | Bronze, Silver layer | 
+| [apply_changes_from_snapshot](https://docs.databricks.com/en/delta-live-tables/python-ref.html#change-data-capture-from-database-snapshots-with-python-in-delta-live-tables) API support | Bronze layer|
+| [append_flow](https://docs.databricks.com/en/delta-live-tables/flows.html#use-append-flow-to-write-to-a-streaming-table-from-multiple-source-streams) API support | Bronze layer|
+| Liquid cluster support | Bronze, Bronze Quarantine, Silver tables|
+| [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/) |  ```databricks labs dlt-meta onboard```, ```databricks labs dlt-meta deploy``` |
+| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using Direct publishing mode |
+
 ## Getting Started
 
 Refer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting_started)

demo/README.md

@@ -198,7 +198,7 @@ This demo will perform following tasks:
 
 6. Run the command 
     ```commandline
-        python demo/launch_silver_fanout_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>>
+        python demo/launch_silver_fanout_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
     ```
 
     - you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.

demo/conf/onboarding.template

@@ -18,6 +18,7 @@
          "cloudFiles.rescuedDataColumn": "_rescued_data",
          "header": "true"
       },
+      "bronze_cluster_by":["customer_id"],
       "bronze_data_quality_expectations_json_prod": "{uc_volume_path}/demo/conf/dqe/customers.json",
       "bronze_database_quarantine_prod": "{uc_catalog_name}.{bronze_schema}",
       "bronze_quarantine_table": "customers_quarantine",
@@ -38,6 +39,7 @@
             "_rescued_data"
          ]
       },
+      "silver_cluster_by":["customer_id"],
       "silver_transformation_json_prod": "{uc_volume_path}/demo/conf/silver_transformations.json",
       "silver_data_quality_expectations_json_prod": "{uc_volume_path}/demo/conf/dqe/customers_silver_dqe.json"
 

demo/launch_dais_demo.py

@@ -2,7 +2,6 @@
 import traceback
 from databricks.sdk.service import jobs, compute
 from src.install import WorkspaceInstaller
-from src.__about__ import __version__
 from integration_tests.run_integration_tests import (
     DLTMETARunner,
     DLTMetaRunnerConf,
@@ -98,7 +97,7 @@ def create_daisdemo_workflow(self, runner_conf: DLTMetaRunnerConf):
             jobs.JobEnvironment(
                 environment_key="dl_meta_int_env",
                 spec=compute.Environment(
-                    client=f"dlt_meta_int_test_{__version__}",
+                    client="1",
                     dependencies=[runner_conf.remote_whl_path],
                 ),
             )

demo/launch_silver_fanout_demo.py

@@ -3,7 +3,6 @@
 import traceback
 from databricks.sdk.service import jobs, compute
 from src.install import WorkspaceInstaller
-from src.__about__ import __version__
 from integration_tests.run_integration_tests import (
     DLTMETARunner,
     DLTMetaRunnerConf,
@@ -105,7 +104,7 @@ def create_sfo_workflow_spec(self, runner_conf: DLTMetaRunnerConf):
             jobs.JobEnvironment(
                 environment_key="dl_meta_int_env",
                 spec=compute.Environment(
-                    client=f"dlt_meta_int_test_{__version__}",
+                    client="1",
                     dependencies=[runner_conf.remote_whl_path],
                 ),
             )

demo/launch_techsummit_demo.py

@@ -27,7 +27,6 @@
 from databricks.sdk.service import jobs, compute
 from dataclasses import dataclass
 from src.install import WorkspaceInstaller
-from src.__about__ import __version__
 from integration_tests.run_integration_tests import (
     DLTMETARunner,
     DLTMetaRunnerConf,
@@ -164,7 +163,7 @@ def create_techsummit_demo_workflow(self, runner_conf: TechsummitRunnerConf):
             jobs.JobEnvironment(
                 environment_key="dl_meta_int_env",
                 spec=compute.Environment(
-                    client=f"dlt_meta_int_test_{__version__}",
+                    client="1",
                     dependencies=[runner_conf.remote_whl_path],
                 ),
             )

docs/config.toml

@@ -1,12 +1,15 @@
 baseURL = 'https://databrickslabs.github.io/dlt-meta/'
 languageCode = 'en-us'
 title = 'DLT-META'
-theme= "hugo-theme-learn"
+theme= "hugo-theme-relearn"
 pluralizeListTitles = false
 canonifyURLs = true
+description = "DLT-META Documentation"
 
 [params]
-    description = "DLT-META Documentation"
-    author = "Ravi Gawai (Databricks)"
+    themeVariant = 'learn'
+    themeVariantAuto = 'learn'
     disableShortcutsTitle = true
-    disableLandingPageButton = false
+    disableLandingPageButton = false
+    author.name = "Ravi Gawai (Databricks)"
+    disableNextPrev = false

docs/content/_index.md

@@ -2,9 +2,9 @@
 title: "DLT-META"
 date: 2021-08-04T14:50:11-04:00
 draft: false
+
 ---
 
-# DLT-META 
 
 ## Project Overview
 DLT-META is a metadata-driven framework designed to work with Databricks Delta Live Tables (DLT). This framework enables the automation of bronze and silver data pipelines by leveraging metadata recorded in an onboarding JSON file. This file, known as the Dataflowspec, serves as the data flow specification, detailing the source and target metadata required for the pipelines.
@@ -40,7 +40,20 @@ In practice, a single generic DLT pipeline reads the Dataflowspec and uses it to
     - Option#1: [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/#dataflow-dlt-pipeline)
     - Option#2: [DLT-META MANUAL](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#dataflow-dlt-pipeline)
 
-
+## DLT-META DLT Features support
+| Features  | DLT-META Support |
+| ------------- | ------------- |
+| Input data sources  | Autoloader, Delta, Eventhub, Kafka, snapshot  |
+| Medallion architecture layers | Bronze, Silver  |
+| Custom transformations | Bronze, Silver layer accepts custom functions|
+| Data Quality Expecations Support | Bronze, Silver layer |
+| Quarantine table support | Bronze layer |
+| [apply_changes](https://docs.databricks.com/en/delta-live-tables/python-ref.html#cdc) API support | Bronze, Silver layer | 
+| [apply_changes_from_snapshot](https://docs.databricks.com/en/delta-live-tables/python-ref.html#change-data-capture-from-database-snapshots-with-python-in-delta-live-tables) API support | Bronze layer|
+| [append_flow](https://docs.databricks.com/en/delta-live-tables/flows.html#use-append-flow-to-write-to-a-streaming-table-from-multiple-source-streams) API support | Bronze layer|
+| Liquid cluster support | Bronze, Bronze Quarantine, Silver tables|
+| [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/) |  ```databricks labs dlt-meta onboard```, ```databricks labs dlt-meta deploy``` |
+| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using Direct publishing mode |
 ## How much does it cost ?
 DLT-META does not have any **direct cost** associated with it other than the cost to run the Databricks Delta Live Tables 
 on your environment.The overall cost will be determined primarily by the [Databricks Delta Live Tables Pricing] (https://databricks.com/product/delta-live-tables-pricing-azure)

docs/content/contributing/onboarding/_index.md

@@ -39,7 +39,7 @@ This demo will perform following tasks:
     ```
 
 6. ```commandline
-    python demo/launch_af_cloudfiles_demo.py --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --dbfs_path=dbfs:/tmp/DLT-META/demo/ --uc_catalog_name=ravi_dlt_meta_uc
+    python demo/launch_af_cloudfiles_demo.py --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --dbfs_path=dbfs:/tmp/DLT-META/demo/ --uc_catalog_name=dlt_meta_uc
     ```
 
 - cloud_provider_name : aws or azure or gcp

docs/content/demo/Append_FLOW_CF.md

@@ -51,7 +51,7 @@ draft: false
 - Following are the mandatory arguments for running EventHubs demo
     - cloud_provider_name: Cloud provider name e.g. aws or azure 
     - dbr_version:  Databricks Runtime Version e.g. 15.3.x-scala2.12
-    - uc_catalog_name : unity catalog name e.g. ravi_dlt_meta_uc
+    - uc_catalog_name : unity catalog name e.g. dlt_meta_uc
     - dbfs_path: Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines e.g. dbfs:/tmp/DLT-META/demo/ 
     - eventhub_namespace: Eventhub namespace e.g. dltmeta
     - eventhub_name : Primary Eventhubname e.g. dltmeta_demo
@@ -62,7 +62,7 @@ draft: false
     - eventhub_port: Eventhub port
 
 7. ```commandline 
-    python demo/launch_af_eventhub_demo.py --cloud_provider_name=aws --uc_catalog_name=ravi_dlt_meta_uc --eventhub_name=dltmeta_demo --eventhub_name_append_flow=dltmeta_demo_af --eventhub_secrets_scope_name=dltmeta_eventhub_creds --eventhub_namespace=dltmeta --eventhub_port=9093 --eventhub_producer_accesskey_name=RootManageSharedAccessKey --eventhub_consumer_accesskey_name=RootManageSharedAccessKey --eventhub_accesskey_secret_name=RootManageSharedAccessKey --uc_catalog_name=ravi_dlt_meta_uc
+    python demo/launch_af_eventhub_demo.py --cloud_provider_name=aws --uc_catalog_name=dlt_meta_uc --eventhub_name=dltmeta_demo --eventhub_name_append_flow=dltmeta_demo_af --eventhub_secrets_scope_name=dltmeta_eventhub_creds --eventhub_namespace=dltmeta --eventhub_port=9093 --eventhub_producer_accesskey_name=RootManageSharedAccessKey --eventhub_consumer_accesskey_name=RootManageSharedAccessKey --eventhub_accesskey_secret_name=RootManageSharedAccessKey
     ```
 
 ![af_eh_demo.png](/images/af_eh_demo.png)