Residential Sales Transaction - ETL Process Using Python

Introduction

The project aim is to develop an END-to-End ETL process. The resulting data will used to apply machine learning models.

Data Source

PSRA - Property Services Regulatory Authority

├── extract                              
└────── extract.py                       
├── transform                       
├────── tranform_handler.py         
└────── transform.py                
├── load                            
└────── load.py                     
├── aws_handler                         
└────── aws_notification_handler.py 
└────── aws_s3_handler.py           
├── schedule_handler                
└────── safe_schedule.py            
└────── scheduler.py                
├── static                          
└──────  css                            
└────── js                          
└────── vendor                          
├── templates                       
└────── base.html                      
└────── index.html                  
├── route.py                            
├── configuration.py                
├── exception.py                    
├── logconfig.py                    
├── Dockerfile                      
├── requirements.txt                    
└── README.md      

Other directories are pytest, output, logs

Technologies

• Python
• Flask
• AWS - S3, SNS
• Docker
• VSCode Editor
• Postman - Simulation
• Jupyter Notebook

Requirements

• boto3
• Flask
• Docker
• pandas

Tasks

• Project Structure
• Project Configuration
• Logging Configuration
• AWS Transaction Handler
• Global Exception Handler
• Extract
• Transform
• Load
• Containerization - DOCKER
• EC2 Deployment using Docker
• S3 Bucket Event Listener
• Enable SNS HTTP endpoint
• CI/CD Pipeline
• Documentation using Sphinx

References

Docker - Youtube - Channel: SelfTuts

Docker - AWS Container

Flask (HTML Template) - Youtube - Channel: freeCodeCamp.org

Web Form Template Design - Colorlib

NOTE: Object Oriented Programming best practices are not followed instead some of the JAVA project design principle are implemented