Skip to content
/ 261-fp Public

We leverage Databricks' MLLib and PySpark libraries to predict continuous outliers using time-series flight delays, NOAA, and geospatial data. We use Yeo-Johnson, Euclidean distance, and custom joins to build a robust dataset—prior to model training. Work completed as part of ML at Scale, DATASCI 261.

0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

JH-UCB/261-fp

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

39 Commits
60 Month Production Preprocessing Run .html
60 Month Production Preprocessing Run .html
 
 
84 Month Production Preprocessing Run .html
84 Month Production Preprocessing Run .html
 
 
Baseline Model v2 Update.png
Baseline Model v2 Update.png
 
 
ES_v2_D_D_T Linear Regression_FINAL.html
ES_v2_D_D_T Linear Regression_FINAL.html
 
 
ES_v2_D_D_T RF GBT.html
ES_v2_D_D_T RF GBT.html
 
 
FP3 LR Modeling Pipeline.png
FP3 LR Modeling Pipeline.png
 
 
FP3 Pipeline.png
FP3 Pipeline.png
 
 
GBM 50k Actual vs YeoJohnson Predicted Dep Delays Extended.png
GBM 50k Actual vs YeoJohnson Predicted Dep Delays Extended.png
 
 
GBM 50k Samples Actual vs YeoJohnson Predicted Dep Delay .png
GBM 50k Samples Actual vs YeoJohnson Predicted Dep Delay .png
 
 
GBM 50k Samples YeoJohnson vs Actual Dep Delay.png
GBM 50k Samples YeoJohnson vs Actual Dep Delay.png
 
 
GBM Clustering of Actual vs Predicted Departure Delays.png
GBM Clustering of Actual vs Predicted Departure Delays.png
 
 
GBM Distribution of Yeo-Johnson Predictions and Actual Departure Delays.png
GBM Distribution of Yeo-Johnson Predictions and Actual Departure Delays.png
 
 
GBM Histogram of Yeo-Johnson Logged Absolute Errors.png
GBM Histogram of Yeo-Johnson Logged Absolute Errors.png
 
 
GBM Residuals vs Actual Departure Delays Clustered.png
GBM Residuals vs Actual Departure Delays Clustered.png
 
 
GBMHistogram of Yeo-Johnson Absolute Errors.png
GBMHistogram of Yeo-Johnson Absolute Errors.png
 
 
LR 50k Actual vs YeoJohnson Predicted Dep Delays Extended.png
LR 50k Actual vs YeoJohnson Predicted Dep Delays Extended.png
 
 
LR 50k Samples Actual vs YeoJohnson Predicted Dep Delay .png
LR 50k Samples Actual vs YeoJohnson Predicted Dep Delay .png
 
 
LR 50k Samples YeoJohnson vs Actual Dep Delay.png
LR 50k Samples YeoJohnson vs Actual Dep Delay.png
 
 
LR Clustering of Actual vs Predicted Departure Delays.png
LR Clustering of Actual vs Predicted Departure Delays.png
 
 
LR Distribution of Yeo-Johnson Predictions and Actual Departure Delays.png
LR Distribution of Yeo-Johnson Predictions and Actual Departure Delays.png
 
 
LR Histogram of Yeo-Johnson Absolute Errors.png
LR Histogram of Yeo-Johnson Absolute Errors.png
 
 
LR Histogram of Yeo-Johnson Logged Absolute Errors.png
LR Histogram of Yeo-Johnson Logged Absolute Errors.png
 
 
LR Residuals vs Actual Departure Delays Clustered.png
LR Residuals vs Actual Departure Delays Clustered.png
 
 
Min-Max Scaled Feature Boxplot .png
Min-Max Scaled Feature Boxplot .png
 
 
PageRank Preprocessing Notebook.html
PageRank Preprocessing Notebook.html
 
 
Pipeline Block Diagram phase 22.drawio.png
Pipeline Block Diagram phase 22.drawio.png
 
 
Pipeline Block Diagram w Color.jpg
Pipeline Block Diagram w Color.jpg
 
 
Prophet Time Series Forecasting Notebook .html
Prophet Time Series Forecasting Notebook .html
 
 
README.md
README.md
 
 
RF 50k Actual vs YeoJohnson Predicted Dep Delays Extended.png
RF 50k Actual vs YeoJohnson Predicted Dep Delays Extended.png
 
 
RF 50k Samples Actual vs YeoJohnson Predicted Dep Delay .png
RF 50k Samples Actual vs YeoJohnson Predicted Dep Delay .png
 
 
RF 50k Samples YeoJohnson vs Actual Dep Delay.png
RF 50k Samples YeoJohnson vs Actual Dep Delay.png
 
 
RF Clustering of Actual vs Predicted Departure Delays.png
RF Clustering of Actual vs Predicted Departure Delays.png
 
 
RF Distribution of Yeo-Johnson Predictions and Actual Departure Delays.png
RF Distribution of Yeo-Johnson Predictions and Actual Departure Delays.png
 
 
RF Histogram of Yeo-Johnson Absolute Errors.png
RF Histogram of Yeo-Johnson Absolute Errors.png
 
 
RF Histogram of Yeo-Johnson Logged Absolute Errors.png
RF Histogram of Yeo-Johnson Logged Absolute Errors.png
 
 
RF Residuals vs Actual Departure Delays Clustered.png
RF Residuals vs Actual Departure Delays Clustered.png
 
 
Screenshot 2024-12-14 at 16.20.33.png
Screenshot 2024-12-14 at 16.20.33.png
 
 
Screenshot 2024-12-15 at 23.43.45.png
Screenshot 2024-12-15 at 23.43.45.png
 
 
Screenshot 2024-12-15 at 23.43.54.png
Screenshot 2024-12-15 at 23.43.54.png
 
 
Screenshot 2024-12-15 at 23.55.29.png
Screenshot 2024-12-15 at 23.55.29.png
 
 
Team 2-3 Phase 3 Report FINAL.html
Team 2-3 Phase 3 Report FINAL.html
 
 
fp3 15 min increments true distribution.png
fp3 15 min increments true distribution.png
 
 
fp3 44 class distribution.png
fp3 44 class distribution.png
 
 
fp3 Delay Groups.png
fp3 Delay Groups.png
 
 
fp3 Histogram of Actual Departure Delays.png
fp3 Histogram of Actual Departure Delays.png
 
 
fp3 Histogram of Engineered DEP_DELAY_NEW.png
fp3 Histogram of Engineered DEP_DELAY_NEW.png
 
 
fp3 Histogram of Log Transformed DEP DELAY NEW.png
fp3 Histogram of Log Transformed DEP DELAY NEW.png
 
 
fp3 Histogram of Yeo-Johnson DEP DELAY.png
fp3 Histogram of Yeo-Johnson DEP DELAY.png
 
 
fp3 actual dep delay boxplot.png
fp3 actual dep delay boxplot.png
 
 
fp3 comparison boxplots.png
fp3 comparison boxplots.png
 
 
fp3 engineered dep delay boxplot.png
fp3 engineered dep delay boxplot.png
 
 
fp3 experiments table.png
fp3 experiments table.png
 
 
fp3 logged actual dep delay boxplot.png
fp3 logged actual dep delay boxplot.png
 
 
fp3 logged engineered dep delay.png
fp3 logged engineered dep delay.png
 
 
fp3 pagerank scatterplot.png
fp3 pagerank scatterplot.png
 
 
fp3 preprocessing markmap.png
fp3 preprocessing markmap.png
 
 
fp3 preprocessing pipeline markmap.html
fp3 preprocessing pipeline markmap.html
 
 
fp3 preprocessing pipeline markmap.svg
fp3 preprocessing pipeline markmap.svg
 
 
fp3 yeo-johnson transformed dep delay.png
fp3 yeo-johnson transformed dep delay.png
 
 
phase 1 credit assignment table .png
phase 1 credit assignment table .png
 
 
phase 2 block CV diagram.png
phase 2 block CV diagram.png
 
 
phase 2 credit assignment pt 1.png
phase 2 credit assignment pt 1.png
 
 
phase 2 credit assignment pt 2.png
phase 2 credit assignment pt 2.png
 
 
phase 2 project plan.png
phase 2 project plan.png
 
 
phase leader table.png
phase leader table.png
 
 
yuge.png
yuge.png
 
 

Repository files navigation

261-fp

Repository containing source figures and reports for predicting minute-by-minute flight delays using time series data, machine learning, data transformations, and PySpark on Databricks. Work completed as part of Machine Learning at Scale, DATASCI 261.

About

We leverage Databricks' MLLib and PySpark libraries to predict continuous outliers using time-series flight delays, NOAA, and geospatial data. We use Yeo-Johnson, Euclidean distance, and custom joins to build a robust dataset—prior to model training. Work completed as part of ML at Scale, DATASCI 261.

Topics

spark distributed-computing databricks time-series-forecasting yeo-johnson-transformations

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages