feat: Modernize demand forecasting solution to 2025 standards

[calreynolds]

🚀 LLM GENERATED:

Complete modernization of retail demand forecasting repository to align with databricks-industry-solutions-blueprints template and 2025 best practices.

Key Modernizations:

🏗️ Architecture & Infrastructure

• Migrated to pure serverless compute (removed all cluster configurations)
• Implemented Unity Catalog for enterprise data governance
• Added Databricks Asset Bundle configuration (databricks.yml)
• Integrated synthetic data generation (eliminated external dependencies)

📊 Repository Restructuring

• Split monolithic script into 3 logical notebooks:
  • 01_data_generation_setup.py - Unity Catalog setup & synthetic data
  • 02_model_training_forecasting.py - Distributed Prophet training
  • 03_results_analysis_visualization.py - Executive insights & KPIs
• Removed R dependencies (SparklyR and SparkR implementations)
• Added GitHub Actions workflow for automated deployment

🔧 Technical Improvements

• Dynamic library installation for serverless compatibility (%pip install)
• Robust schema definitions to prevent Delta merge conflicts
• Comprehensive error handling and progress tracking
• Eliminated caching operations incompatible with serverless

🧹 Code Quality & Cleanup

• Removed legacy files and streamlined directory structure
• Added proper documentation and environment configuration
• Implemented template-compliant file organization
• Added comprehensive README with business-first messaging

This modernization delivers an enterprise-ready, industry-focused solution that demonstrates Databricks' retail AI capabilities while following 2025 best practices.

[Copilot]

Pull Request Overview

Modernize the demand forecasting solution to use serverless compute, Unity Catalog, and Databricks Asset Bundles while restructuring notebooks, removing R dependencies, and introducing CI/CD.

• Migrated infrastructure to serverless compute and Unity Catalog with asset bundle configuration.
• Refactored monolithic code into three logical Python notebooks and removed legacy R notebooks.
• Added GitHub Actions CI/CD workflow, synthetic data generation, and updated documentation.

Reviewed Changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/cleanup.sh	New cleanup script to remove deployed Databricks resources.
notebooks/01_data_generation_setup.py	Notebook for Unity Catalog setup and synthetic data creation.
notebooks/02_model_training_forecasting.py	Notebook for distributed Prophet model training.
notebooks/03_results_analysis_visualization.py	Notebook for executive dashboards and visualizations.
env.example	Environment configuration template.
databricks.yml	Databricks Asset Bundle configuration.
README.md	Updated README to reflect new structure and features.
.github/workflows/deploy.yml	CI/CD pipeline for bundle validation and deployment.

Comments suppressed due to low confidence (3)

.github/workflows/deploy.yml:99

• The job key demand_forecasting_pipeline does not match the defined demand_forecasting_workflow in databricks.yml. Update the command to use the correct job key.

          databricks bundle run demand_forecasting_pipeline --target dev

README.md:60

• The README references demand_forecasting_pipeline.ipynb, but the repository contains separate 01_, 02_, and 03_ notebook files. Update this path to reflect the current notebook structure.

│   └── demand_forecasting_pipeline.ipynb  # Main forecasting notebook

notebooks/02_model_training_forecasting.py:357

• The code references forecasts instead of the defined forecast_df, causing a NameError. It should use forecast_df.select(...).

    unique_combinations = forecasts.select("store", "item").distinct().count()