This project explores an effective way to identify misinformation by using sophisticated big data solution, including Apache Spark, Apache Kafka, and Databricks. It focuses on processing vast, ongoing data streams, integrating real-time RSS feeds with static news databases. The system enhances its efficiency through high-level preprocessing methods such as normalization and tokenization, along with effective data management practices that reduce computational demands. The approach combines various modeling techniques such as global, local, and ensemble—to achieve a well rounded performance, with an accuracy, precision, and recall rate of about 72.5% and an F1 score of 72.3%.
The model code is in 'databricks/model' The kafka code is in 'databricks/kafka' The data ingestion code is in 'databricks/data-pipeline'