README.md

Project Overview

This project aims to tackle the challenging task of Deepfake detection using state-of-the-art techniques. The project integrates robust image and video analysis, leveraging pre-trained models, data augmentation, and novel architectures to detect manipulated and synthetic Deepfake content. By employing a combination of HDDM (Hybrid Deepfake Detection Mechanism) and GRU (Gated Recurrent Units), the system targets improved accuracy and robustness for real-world applications.

Features:

1. Detection of manipulated and synthetic Deepfake images and videos.
2. Use of advanced data augmentation techniques to enhance dataset diversity.
3. Integration of HDDM and GRU models for hybrid detection.
   • HDDM constructs unique textures from facial images using CNN to assess naturalness and detect manipulated and synthetic Deepfake attacks.
   • GRU is incorporated to handle temporal dependencies in video frames, improving the detection of video-based Deepfake content.
4. Evaluation metrics include confusion matrices, training loss, accuracy plots, and classification reports.

---

Folder Structure

The following outlines the folder and file structure of this project:

Project/
│  
├──HDDM_Project/
│   ├── data/
│   │   └── Contains pre-processed datasets for image-based detection.
│   ├── data_video/
│   │   ├── fake/
│   │   ├── real/
│   │   └── Extracted frames from video-based datasets for detection.
│   └── video/
│       ├── fake/
│       └── real/
│           └── Original videos for processing.
│
├── models/
│   ├── hddm_model_1.pth        # Pre-trained HDDM model.
│   ├── hddm_model_finetuned.pth # Fine-tuned HDDM model.
│   ├── gru_model_scratch.pth   # GRU model trained from scratch.
│   └── Corresponding model scripts.
├── scripts/
│   ├── ROI_image.py            # Region of Interest extraction for images.
│   ├── ROI_video.py            # Region of Interest extraction for videos.
│   ├── Dataset_preprocess.py   # Dataset pre-processing script.
│   ├── Download_Datasets.py    # Script to download datasets.
│   └── train_GRU.ipynb         # Training notebook for GRU and HDDM integration.
│   └── train_HDDM.ipynb        # Training notebook for HDDM model only.
├── confusion_matrix.png         # Confusion matrix plot for evaluation.
├── README.md                    # Project documentation.
└── shape_predictor_68_face_landmarks.dat # Dlib face landmark model.

---

References

Data Sources

1. Fake/Real Image Dataset

   The dataset can be accessed via the following reference:

   @misc{song2023robustness,
         title={Robustness and Generalizability of Deepfake Detection: A Study with Diffusion Models},
         author={Haixu Song and Shiyu Huang and Yinpeng Dong and Wei-Wei Tu},
         year={2023},
         eprint={2309.02218},
         archivePrefix={arXiv},
         primaryClass={cs.CV}
   }
   

   Repository Link: Robustness and Generalizability of Deepfake Detection

2. Fake/Real Video Dataset

   The dataset, "WildDeepfake", can be accessed via the following reference:

   @inproceedings{zi2020wilddeepfake,
     title={Wilddeepfake: A challenging real-world dataset for deepfake detection},
     author={Zi, Bojia and Chang, Minghao and Chen, Jingjing and Ma, Xingjun and Jiang, Yu-Gang},
     booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
     pages={2382--2390},
     year={2020}
   }
   

   Dataset Link: WildDeepfake

Literature Reference

3. Preserving Manipulated and Synthetic Deepfake Detection

   Chit-Jie Chew, Yu-Cheng Lin, Ying-Chin Chen, Yun-Yi Fan, Jung-San Lee,
   Preserving manipulated and synthetic Deepfake detection through face texture naturalness,
   Journal of Information Security and Applications,
   Volume 83,
   2024,
   103798,
   ISSN 2214-2126,
   https://doi.org/10.1016/j.jisa.2024.103798.
   
   Abstract: With the rapid development of deep learning and face recognition technology, AI(Artificial Intelligence) experts have rated Deepfake cheating as the top AI threat. It is difficult for the human eye to distinguish the fake face images generated by Deepfake. Therefore, it has become a popular tool for criminals to seek benefits. Deepfake can be mainly divided into two types, a manipulated Deepfake that falsifies images of others by targeting real faces, and a synthetic Deepfake using GAN to generate a new fake image. So far, seldom cybersecurity system is able to detect these two types simultaneously. In this article, we aim to propose a hybrid Deepfake detection mechanism (HDDM) based on face texture and naturalness degree. HDDM constructs a unique texture from a facial image based on CNN(Convolutional Neural Network) and builds a naturalness degree recognition model via DNN(Deep Neural Network) to help cheating detection. Experimental results have proved that HDDM possesses a sound effect and stability for synthetic and manipulated Deepfake attacks. In particular, the WildDeepfake simulation has demonstrated the possibility of applying HDDM to the real world.
   
   Keywords: Facial texture; Naturalness; Deepfake; DNN