Weather ETL Pipeline with Apache Airflow

Overview

The Weather ETL Pipeline is designed to automate the extraction, transformation, and loading (ETL) of weather data using Apache Airflow, Docker, and Astronomer. This project retrieves real-time weather information from the Open Meteo API and stores it in a PostgreSQL database for further analysis.

🚀 Project Structure

• dags/: Contains the Python files for Airflow DAGs.
  • weather_etl_pipeline.py: Implements the ETL process by extracting weather data, transforming it, and loading it into the database.
• Dockerfile: Specifies the versioned Astro Runtime Docker image for executing the project.
• requirements.txt: Lists the Python packages required for the project.
• plugins/: Directory for custom or community plugins.
• airflow_settings.yaml: Local configuration for Airflow Connections, Variables, and Pools.

🛠️ Requirements

• Python: 3.9+
• Apache Airflow: 2.5.0
• PostgreSQL: Required for data storage

🔧 Installation Instructions

1. Clone the Repository:

    git clone https://github.com/abu14/ETL-Weather-Airflow.git
    cd ETL-Weather-Airflow

2. Build Docker Image

   docker build -t weather-etl-pipeline .

4. Start Airflow

  docker-compose up

📌 Usage

• Access the Airflow web interface at http://localhost:8080.
• Trigger the weather_etl_pipeline DAG to initiate the ETL process.

License

This project is licensed under the MIT License. See the LICENSE file for details.