Add NewsAPI integration and update environment configuration

- Introduced scripts for extracting news articles related to known entities using NewsAPI.
- Updated .env.example to include NEWSAPI_KEY.
- Modified env.py to include newsapi_key in the database configuration.
- Adjusted README.md to reflect local processing and new extraction scripts.
- Removed unused Docker services from docker-compose.yml.

Author: ByteMeDirk

.env.example

@@ -4,3 +4,5 @@ DB_PORT=5432
 DB_NAME=postgres
 DB_USER=postgres
 DB_PASSWORD=postgres
+# API configuration
+NEWSAPI_KEY=your_newsapi_key_here

README.md

@@ -1,6 +1,6 @@
 # Semantic Medallion Data Platform
 
-A modern data platform implementing the medallion architecture on Google Cloud Platform.
+A modern data platform implementing the medallion architecture with local processing.
 
 ## Architecture Overview
 
@@ -13,24 +13,25 @@ This project implements a medallion architecture for data lakes, which organizes
 ## Tech Stack
 
 - **Data Processing**: PySpark, Delta Lake
-- **Cloud Infrastructure**: Google Cloud Platform (GCS, BigQuery)
+- **Database**: PostgreSQL
 - **Orchestration**: Prefect
 - **Transformation**: dbt
 - **Data Quality**: Great Expectations
 - **Local Development**: Docker, Poetry
-- **Infrastructure as Code**: Terraform
+- **External APIs**: NewsAPI
 
 ## Project Structure
 
 ```
 semantic-medallion-data-platform/
 ├── .github/                      # GitHub Actions workflows
+├── data/                         # Data files
+│   └── known_entities/           # Known entities data files
 ├── docs/                         # Documentation
-├── infrastructure/               # Terraform configurations
-│   ├── environments/             # Environment-specific configurations
-│   └── modules/                  # Reusable Terraform modules
 ├── semantic_medallion_data_platform/  # Main package
 │   ├── bronze/                   # Bronze layer processing
+│   │   ├── brz_01_extract_newsapi.py        # Extract news articles from NewsAPI
+│   │   └── brz_01_extract_known_entities.py # Extract known entities from CSV files
 │   ├── silver/                   # Silver layer processing
 │   ├── gold/                     # Gold layer processing
 │   ├── common/                   # Shared utilities
@@ -49,8 +50,6 @@ semantic-medallion-data-platform/
 - Python 3.9+
 - [Poetry](https://python-poetry.org/docs/#installation)
 - [Docker](https://docs.docker.com/get-docker/)
-- [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)
-- Google Cloud account with appropriate permissions
 
 ### Installation
 
@@ -75,39 +74,61 @@ semantic-medallion-data-platform/
    cp .env.example .env
    ```
 
-   Edit the `.env` file to set your database credentials and other environment variables.
+   Edit the `.env` file to set your database credentials and other environment variables. Make sure to set your NewsAPI key if you plan to use the news article extraction functionality:
+   ```
+   NEWSAPI_KEY=your_newsapi_key_here
+   ```
+
+   You can obtain a NewsAPI key by signing up at [https://newsapi.org/](https://newsapi.org/).
 
 ### Local Development
 
 Start the local development environment:
 
 ```bash
+cd docker
 docker-compose up -d
 ```
 
 This will start:
 - Local PostgreSQL database
-- Local GCS emulator
-- Other required services
 
 ### Running Tests
 
 ```bash
 poetry run pytest
 ```
 
-### Deploying to GCP
+### Running Bronze Layer Processes
 
-1. Initialize Terraform:
-   ```bash
-   cd infrastructure/environments/dev
-   terraform init
-   ```
+#### Extracting News Articles from NewsAPI
+
+To extract news articles for known entities:
+
+```bash
+cd semantic-medallion-data-platform
+python -m semantic_medallion_data_platform.bronze.brz_01_extract_newsapi --days_back 7
+```
+
+This will:
+1. Fetch known entities from the database
+2. Query NewsAPI for articles mentioning each entity
+3. Store the articles in the bronze.newsapi table
+
+#### Extracting Known Entities
+
+To load known entities from CSV files:
+
+```bash
+cd semantic-medallion-data-platform
+python -m semantic_medallion_data_platform.bronze.brz_01_extract_known_entities --raw_data_filepath data/known_entities/
+```
+
+This will:
+1. Read entity data from CSV files in the specified directory
+2. Process and transform the data
+3. Store the entities in the bronze.known_entities table
 
-2. Apply Terraform configuration:
-   ```bash
-   terraform apply
-   ```
 
 ## Contributing
 

data/known_entities/known_entites_org.csv

@@ -0,0 +1,40 @@
+entity_name,entity_type,entity_description
+Microsoft,ORG,"Multinational technology corporation co-founded by Bill Gates, specializing in software, cloud computing, and consumer electronics."
+Apple,ORG,"Multinational technology company co-founded by Steve Jobs, known for consumer electronics, software, and digital services."
+Tesla,ORG,"Electric vehicle and clean energy company founded and led by Elon Musk, pioneering sustainable transportation."
+SpaceX,ORG,"Private space exploration company founded by Elon Musk, developing spacecraft and satellite internet services."
+Meta,ORG,"Social media and technology conglomerate founded by Mark Zuckerberg, formerly known as Facebook."
+Facebook,ORG,"Social networking platform founded by Mark Zuckerberg, now part of Meta's ecosystem."
+Google,ORG,"Search engine and technology company co-founded by Larry Page and Sergey Brin, now part of Alphabet."
+Alphabet,ORG,"Parent company of Google led by Sundar Pichai, encompassing various technology ventures and moonshot projects."
+Amazon,ORG,"E-commerce and cloud computing giant founded by Jeff Bezos, dominating online retail and AWS services."
+Dell Technologies,ORG,"Computer technology company founded by Michael Dell, specializing in personal computers and enterprise solutions."
+Oracle,ORG,"Database software corporation co-founded by Larry Ellison, leading enterprise software and cloud computing."
+Nvidia,ORG,"Graphics processing and AI chip company led by Jensen Huang, powering modern AI and gaming."
+Salesforce,ORG,"Cloud-based customer relationship management platform founded by Marc Benioff."
+Netflix,ORG,"Streaming entertainment service co-founded by Reed Hastings, revolutionizing media consumption."
+Twitter,ORG,"Social media platform co-founded by Jack Dorsey, now known as X under Elon Musk's ownership."
+X,ORG,"Social media platform formerly known as Twitter, acquired and rebranded by Elon Musk."
+Uber,ORG,"Ride-sharing and mobility platform co-founded by Travis Kalanick, transforming transportation services."
+Airbnb,ORG,"Home-sharing marketplace co-founded by Brian Chesky, disrupting the hospitality industry."
+Spotify,ORG,"Music streaming platform co-founded by Daniel Ek, leading digital music distribution."
+Snapchat,ORG,"Multimedia messaging app co-founded by Evan Spiegel, pioneering ephemeral content sharing."
+Dropbox,ORG,"Cloud storage service co-founded by Drew Houston, simplifying file sharing and collaboration."
+Stripe,ORG,"Online payment processing company co-founded by Patrick and John Collison, powering internet commerce."
+LinkedIn,ORG,"Professional networking platform co-founded by Reid Hoffman, connecting business professionals worldwide."
+PayPal,ORG,"Digital payment system co-founded by Peter Thiel and Elon Musk, enabling online financial transactions."
+Palantir,ORG,"Data analytics company co-founded by Peter Thiel, specializing in big data analysis for enterprises."
+Asana,ORG,"Work management platform co-founded by Dustin Moskovitz, helping teams organize and track projects."
+Instagram,ORG,"Photo and video sharing platform co-founded by Kevin Systrom, acquired by Facebook/Meta."
+Neuralink,ORG,"Neurotechnology company founded by Elon Musk, developing brain-computer interface technology."
+The Boring Company,ORG,"Tunnel construction company founded by Elon Musk, creating underground transportation systems."
+YouTube,ORG,"Video sharing platform formerly led by Susan Wojcicki, owned by Google/Alphabet."
+Square,ORG,"Financial services company co-founded by Jack Dorsey, providing payment solutions for businesses."
+Pixar,ORG,"Animation studio acquired by Steve Jobs, creating computer-animated films before Disney acquisition."
+NeXT,ORG,"Computer company founded by Steve Jobs after leaving Apple, later acquired by Apple."
+Beats Electronics,ORG,"Audio products company acquired by Apple under Tim Cook's leadership for $3 billion."
+WhatsApp,ORG,"Messaging application acquired by Facebook/Meta, enabling global instant communication."
+Threads,ORG,"Social networking platform launched by Meta under Mark Zuckerberg's leadership."
+Cascade Investment,ORG,"Private investment firm founded and controlled by Bill Gates, managing his wealth and investments."
+Breakthrough Energy Ventures,ORG,"Clean energy investment fund founded by Bill Gates, backing climate change solutions."
+TerraPower,ORG,"Nuclear reactor design company co-founded by Bill Gates, developing

data/known_entities/known_entities_loc.csv

@@ -0,0 +1,50 @@
+entity_name,entity_type,entity_description
+Medina,LOC,"Exclusive city in Washington state near Lake Washington, home to tech billionaires including Bill Gates."
+Seattle,LOC,"Major city in Washington state, birthplace of Bill Gates and headquarters region for Microsoft."
+Los Altos,LOC,"City in California's Silicon Valley, location of Steve Jobs' garage where Apple was founded."
+Cupertino,LOC,"City in California, headquarters of Apple Inc. and center of consumer electronics innovation."
+Palo Alto,LOC,"City in California's Silicon Valley, hub for tech startups and venture capital firms."
+Menlo Park,LOC,"City in California, headquarters of Meta/Facebook and numerous venture capital firms."
+Mountain View,LOC,"City in California, headquarters of Google/Alphabet and center of search technology."
+Redmond,LOC,"City in Washington state, headquarters of Microsoft Corporation."
+Austin,LOC,"Capital city of Texas, headquarters of Dell Technologies and Oracle Corporation."
+Round Rock,LOC,"City in Texas, location of Dell's primary headquarters and manufacturing facilities."
+Los Altos Hills,LOC,"Affluent city in California, home to tech executives including Sundar Pichai and Jensen Huang."
+Atherton,LOC,"Wealthy city in California, residence of tech executives including Sheryl Sandberg."
+San Francisco,LOC,"Major city in California, headquarters of numerous tech companies including Salesforce and Twitter/X."
+Fremont,LOC,"City in California, location of Tesla's main manufacturing facility."
+Hawthorne,LOC,"City in California, headquarters of SpaceX and location of rocket development."
+Boca Chica,LOC,"Location in Texas, site of SpaceX's Starship development and launch facility."
+Bastrop,LOC,"City in Texas, location of Elon Musk's corporate campus including X headquarters."
+Santa Clara,LOC,"City in California, birthplace of Susan Wojcicki and center of semiconductor industry."
+Kauai,LOC,"Hawaiian island, location of Mark Zuckerberg's extensive private estate and compound."
+Indian Creek,LOC,"Exclusive island in Miami, Florida, known as 'Billionaire Bunker' and home to Jeff Bezos."
+Manalapan,LOC,"Wealthy town in Florida, location of Larry Ellison's $173 million estate."
+Lanai,LOC,"Hawaiian island owned almost entirely by Larry Ellison as his primary residence."
+Dolores Heights,LOC,"Neighborhood in San Francisco, residence of Airbnb founder Brian Chesky."
+Stanford,LOC,"University and surrounding area in California, where Google founders Larry Page and Sergey Brin met."
+Cambridge,LOC,"City in Massachusetts, location of Harvard University where Mark Zuckerberg founded Facebook."
+Hyderabad,LOC,"City in India, birthplace of Microsoft CEO Satya Nadella."
+Moscow,LOC,"Capital of Russia, birthplace of Google co-founder Sergey Brin."
+Lansing,LOC,"City in Michigan, birthplace of Google co-founder Larry Page."
+Mobile,LOC,"City in Alabama, birthplace of Apple CEO Tim Cook."
+Robertsdale,LOC,"City in Alabama, hometown where Tim Cook grew up."
+Eden,LOC,"City in Utah, location associated with Netflix founder Reed Hastings."
+Costa Rica,LOC,"Central American country, frequent vacation destination of Twitter co-founder Jack Dorsey."
+Silicon Valley,LOC,"Technology hub region in California, center of global tech innovation and venture capital."
+San Jose,LOC,"Major city in California's Silicon Valley, center of technology and innovation."
+Sunnyvale,LOC,"City in California's Silicon Valley, location of numerous tech companies."
+Santa Monica,LOC,"City in California, location of Snapchat headquarters."
+Beverly Hills,LOC,"Affluent city in California, residence area for many tech executives."
+Malibu,LOC,"Coastal city in California, residence area for tech billionaires."
+Newport Beach,LOC,"Wealthy coastal city in California, location of tech executive residences."
+Lake Tahoe,LOC,"Mountain lake region, popular retreat location for Silicon Valley executives."
+Napa Valley,LOC,"Wine region in California, popular investment and residence area for tech billionaires."
+Washington State,LOC,"U.S. state, home to Microsoft headquarters and Bill Gates' residence."
+California,LOC,"U.S. state, center of global technology industry and Silicon Valley."
+Texas,LOC,"U.S. state, emerging tech hub with headquarters of Dell, Oracle, and Tesla operations."
+Florida,LOC,"U.S. state, location of tech billionaire estates and emerging tech scene."
+Hawaii,LOC,"U.S. state, location of tech billionaire private estates and retreats."
+New York City,LOC,"Major U.S. city, location of tech company offices and executive residences."
+Boston,LOC,"Major U.S. city, technology and education hub with MIT and Harvard nearby."
+Chicago,LOC,"Major U.S. city, location where Larry Ellison attended university."

data/known_entities/known_entities_per.csv

@@ -0,0 +1,31 @@
+entity_name,entity_type,entity_description
+Bill Gates,PER,"Co-founder of Microsoft, philanthropist, and pioneer in the software industry."
+Steve Jobs,PER,"Co-founder of Apple Inc., visionary in personal computing and consumer electronics."
+Elon Musk,PER,"CEO of Tesla, SpaceX, Neuralink, and co-founder of PayPal, known for innovation in electric vehicles and space exploration."
+Mark Zuckerberg,PER,"Co-founder and CEO of Meta (Facebook), influential in social media and virtual reality."
+Sundar Pichai,PER,"CEO of Alphabet and Google, key figure in AI and cloud computing."
+Tim Cook,PER,"CEO of Apple Inc., leading innovation in consumer electronics and services."
+Satya Nadella,PER,"CEO of Microsoft, driving cloud computing and AI advancements."
+Michael Dell,PER,"Founder and CEO of Dell Technologies, pioneer in direct sales and customized computing."
+Larry Page,PER,"Co-founder of Google, instrumental in search engine technology and AI development."
+Sergey Brin,PER,"Co-founder of Google, key contributor to search technology and innovation."
+Jeff Bezos,PER,"Founder of Amazon, innovator in e-commerce and cloud computing."
+Sheryl Sandberg,PER,"Former COO of Facebook, influential in social media business strategy."
+Susan Wojcicki,PER,"Former CEO of YouTube, key figure in online video and digital advertising."
+Reed Hastings,PER,"Co-founder and CEO of Netflix, pioneer in streaming media."
+Jack Dorsey,PER,"Co-founder of Twitter and Square, influential in social media and fintech."
+Larry Ellison,PER,"Co-founder and CEO of Oracle Corporation, major figure in database technology."
+Jensen Huang,PER,"CEO of Nvidia, leader in graphics processing and AI chip development."
+Marc Benioff,PER,"CEO of Salesforce, pioneer in cloud-based customer relationship management."
+Travis Kalanick,PER,"Co-founder of Uber, revolutionary in ride-sharing and gig economy."
+Brian Chesky,PER,"Co-founder and CEO of Airbnb, innovator in sharing economy and hospitality."
+Daniel Ek,PER,"Co-founder and CEO of Spotify, leader in music streaming technology."
+Evan Spiegel,PER,"Co-founder and CEO of Snapchat, innovator in ephemeral messaging and AR."
+Drew Houston,PER,"Co-founder and CEO of Dropbox, pioneer in cloud storage solutions."
+Patrick Collison,PER,"Co-founder and CEO of Stripe, leader in online payment processing."
+John Collison,PER,"Co-founder and President of Stripe, innovator in fintech infrastructure."
+Reid Hoffman,PER,"Co-founder of LinkedIn, influential in professional networking and venture capital."
+Peter Thiel,PER,"Co-founder of PayPal and Palantir, prominent venture capitalist and entrepreneur."
+Dustin Moskovitz,PER,"Co-founder of Facebook and Asana, influential in social media and productivity software."
+Bobby Murphy,PER,"Co-founder and CTO of Snapchat, key figure in mobile messaging innovation."
+Kevin Systrom,PER,"Co-founder of Instagram, pioneer in photo-sharing and social media."

data/sources.md

@@ -18,58 +18,5 @@ services:
       timeout: 5s
       retries: 5
 
-  gcs-emulator:
-    image: fsouza/fake-gcs-server
-    container_name: medallion-gcs
-    ports:
-      - "4443:4443"
-    command: ["-scheme", "http", "-port", "4443", "-public-host", "localhost:4443"]
-    volumes:
-      - gcs-data:/storage
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:4443"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-
-  spark-master:
-    image: bitnami/spark:3.4
-    container_name: medallion-spark-master
-    environment:
-      - SPARK_MODE=master
-      - SPARK_RPC_AUTHENTICATION_ENABLED=no
-      - SPARK_RPC_ENCRYPTION_ENABLED=no
-      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
-      - SPARK_SSL_ENABLED=no
-    ports:
-      - "8080:8080"
-      - "7077:7077"
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8080"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-
-  spark-worker:
-    image: bitnami/spark:3.4
-    container_name: medallion-spark-worker
-    environment:
-      - SPARK_MODE=worker
-      - SPARK_MASTER_URL=spark://spark-master:7077
-      - SPARK_WORKER_MEMORY=1G
-      - SPARK_WORKER_CORES=1
-      - SPARK_RPC_AUTHENTICATION_ENABLED=no
-      - SPARK_RPC_ENCRYPTION_ENABLED=no
-      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
-      - SPARK_SSL_ENABLED=no
-    depends_on:
-      - spark-master
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8081"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-
 volumes:
   postgres-data:
-  gcs-data:

docker/docker-compose.yml

@@ -2,7 +2,8 @@
 
 ## Overview
 
-The medallion architecture is a data organization framework that structures data into three distinct layers, each with its own purpose and characteristics:
+The medallion architecture is a data organization framework that structures data into three distinct layers, each with
+its own purpose and characteristics:
 
 1. **Bronze Layer (Raw Data)**
 2. **Silver Layer (Validated Data)**
@@ -15,21 +16,24 @@ This architecture provides a clear separation of concerns and enables efficient
 The Bronze layer contains raw data ingested from various sources with minimal or no transformations.
 
 ### Characteristics:
+
 - Raw, unprocessed data
 - Exact copy of source data
 - Immutable
 - Append-only
 - Full history preserved
 
 ### Storage:
-- GCS bucket: `{project_id}-bronze`
-- BigQuery dataset: `bronze`
+
+- Local filesystem: `data/bronze`
+- PostgreSQL schema: `bronze`
 
 ## Silver Layer
 
 The Silver layer contains cleansed, validated, and transformed data that is ready for analysis.
 
 ### Characteristics:
+
 - Validated and cleansed data
 - Standardized schemas
 - Data quality checks applied
@@ -38,23 +42,27 @@ The Silver layer contains cleansed, validated, and transformed data that is read
 - Business keys established
 
 ### Storage:
-- GCS bucket: `{project_id}-silver`
-- BigQuery dataset: `silver`
+
+- Local filesystem: `data/silver`
+- PostgreSQL schema: `silver`
 
 ## Gold Layer
 
-The Gold layer contains business-level aggregates and metrics that are ready for consumption by end-users and applications.
+The Gold layer contains business-level aggregates and metrics that are ready for consumption by end-users and
+applications.
 
 ### Characteristics:
+
 - Business-level aggregates
 - Denormalized for query performance
 - Optimized for specific use cases
 - Ready for consumption
 - Often includes dimensional models
 
 ### Storage:
-- GCS bucket: `{project_id}-gold`
-- BigQuery dataset: `gold`
+
+- Local filesystem: `data/gold`
+- PostgreSQL schema: `gold`
 
 ## Data Flow