Reinforced Embodied Active Defense: Exploiting Adaptive Interaction for Robust Visual Perception in Adversarial 3D Environments

🚀 Overview

Reinforced Embodied Active Defense (Rein-EAD) is a proactive defense framework that leverages adaptive exploration and interaction with 3D environments to improve perception robustness against adversarial attacks. Unlike passive defense methods, our framework proactively explores the environment to gather more robust visual information for decision-making. This repository contains the official implementation of both Rein-EAD and its predecessor EAD (Embodied Active Defense).

Key Features

• ​Adaptive Interaction​: Dynamic exploration strategies for robust perception
• ​Policy Learning​:integrates multi-step accumulative interactions and policy learning into a cohesive framework.
• ​Strong Empirical Results​: superior effectiveness over state-of-the-art passive defenses, strong generalization against various unseen and adaptive attacks, and adaptability to complex real-world scenarios.

📋 Table of Contents

• Installation
• Getting Started
• Dataset Preparation
• Model Checkpoints
• Evaluation
• Training
• Baseline Methods
• Citation
• Acknowledgments

🔧 Installation

git clone https://github.com/thu-ml/EmbodiedActiveDefense.git
cd EmbodiedActiveDefense

conda create -n ead38 python=3.8
conda activate ead38
pip install -r requirements.txt

🎯 Getting Started

Directory Structure

Organize your project directory as follows:

EmbodiedActiveDefense/
├── EmbodiedActiveDefense/      # Main codebase
└── EAD_dataset/                # Data and checkpoints
    ├── ckpt/                   # Model checkpoints
    │   ├── backbone/
    │   │   └── ms1mv3_arcface_r50_fp16.pth
    │   ├── eg3d/
    │   │   └── ffhq512-128.pkl
    │   ├── ead/
    │   │   └── ead_finetune.pth.tar
    │   └── rein_ead/
    │       ├── rein_ead.pth
    │       └── results.json
    ├── datasets/               # Training and evaluation data
    │   ├── face_dataset/
    │   ├── adv_dataset/
    │   ├── offline_face_dataset/
    │   ├── OAPA_mini/
    │   ├── OAPA/
    │   ├── aligned_CelebA_2000_2000.txt
    │   ├── front_CelebA_dodging_100.txt
    │   ├── front_CelebA_impersonation_100.txt
    │   └── identity_2500.json
    └── face_recon/             # Face reconstruction components
        ├── BFM/
        ├── masks/
        └── mask_landmark.txt

📊 Dataset Preparation

Our framework is built upon two key components:

Core Dependencies

1. ​Deep3DFaceRecon: For 3D face reconstruction
2. ​EG3D: For 3D-aware image synthesis

Required Downloads

Component	Description	Source
ffhq512-128.pkl	EG3D pre-trained model	EG3D Models
BFM	Basel Face Model	Deep3DFaceRecon
Face Dataset	Evaluation faces	face_dataset_archive.tar.gz
Adversarial Dataset	Evaluation adversarial examples	adv_dataset.tar.gz
Offline Face Dataset	Pre-training data	offline_face_dataset_archive.tar.gz
OAPA Dataset	Offline Adversarial Patch Approximation	OAPA.tar.gz

🏋️ Model Checkpoints

Pre-trained Models

Model	Description	Download
Rein-EAD	Our reinforcement learning enhanced model	rein_ead.pth
EAD	Base embodied active defense model	ead_finetune.pth.tar
ArcFace Backbone	Face recognition backbone	ms1mv3_arcface_r50_fp16.pth

🔍 Evaluation

Evaluate our models against various adversarial attacks including MIM, EOT, Face3dAdv, RGF, and N_attack under both dodging and impersonation scenarios.

Evaluate Rein-EAD

bash scripts/eval_rein_ead.sh

Evaluate EAD

bash scripts/eval_ead.sh

Attack Types Supported

Attack Method	Description
​MIM ​	Momentum Iterative Method
​EOT ​	Expectation Over Transformations
Face3dAdv​	3D Face Adversarial Attack
RGF​	Random Gradient-Free Attack
​N_attack​	Normal Distribution Attack

🎓 Training

​Note​: Training requires EG3D simulation environment running online. We recommend using multiple GPUs for acceleration.

Training Rein-EAD

1. Pre-training

bash scripts/pretrain_rein_ead.sh

2. PPO Fine-tuning

bash scripts/ppo_finetune_rein_ead.sh

Training EAD

1. Pre-training

bash scripts/pretrain_ead.sh

2. Differentiable Fine-tuning

bash scripts/diff_finetune_ead.sh

🔬 Baseline Methods

We compare against several state-of-the-art defense methods:

Method	Description	Repository
LGS	Local Gradients Smoothing	fabiobrau/local_gradients_smoothing
SAC	Segment and Complete	joellliu/SegmentAndComplete
PatchZero	Patch Detection and Zeroing	Elysia200207/PatchZero
DOA	Defense against Occlusion Attacks	tongwu2020/phattacks

📝 Citation

If you find this work useful in your research, please consider citing:

@article{yang2025reinforced,
  title={Reinforced Embodied Active Defense: Exploiting Adaptive Interaction for Robust Visual Perception in Adversarial 3D Environments},
  author={Yang, Xiao and Wu, Lingxuan and Wang, Lizhong and Ying, Chengyang and Su, Hang and Zhu, Jun},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2025},
  publisher={IEEE}
}

@article{wu2024ead,
  title={Embodied Active Defense: Leveraging Recurrent Feedback to Counter Adversarial Patches},
  author={Wu, Lingxuan and Yang, Xiao and Dong, YinPeng and Xie, Liuwei and Su, Hang and Zhu, Jun},
  journal={arXiv preprint arXiv:2404.00540},
  url={https://arxiv.org/abs/2404.00540},
  year={2024}
}

🙏 Acknowledgments

We thank the authors of the following projects that made this work possible:

• EG3D for the 3D-aware synthesis framework
• Deep3DFaceRecon for 3D face reconstruction
• InsightFace for face recognition models

📧 Contact

For questions and discussions, please open an issue or contact the authors.