The first adversarial attack method specifically designed for person segmentation networks
FashionAdv introduces a novel approach to adversarial attacks on person segmentation networks by synthesizing natural-looking clothing textures that can make persons invisible to deep learning models. Unlike traditional adversarial attacks that modify entire images, FashionAdv strategically targets only clothing regions, maintaining image quality while achieving robust performance against various transformations.
Fashion-Guided Adversarial Attack on Person Segmentation
CVPR 2021 Workshop
This paper presents the first adversarial example-based method for attacking human instance segmentation networks. Our approach automatically identifies attackable regions and generates natural-looking adversarial textures that can make persons invisible to segmentation networks while maintaining high image quality.
- π¨ Fashion-Guided: Leverages fashion style images to generate natural-looking adversarial textures
- π― Targeted Attack: Focuses only on clothing regions, minimizing impact on overall image quality
- π‘οΈ Robust: Resistant to JPEG compression, image filtering, and various transformations
- ποΈ Imperceptible: Changes are inconspicuous to human observers
- π SOTA Performance: Significantly outperforms conventional adversarial methods
| Method | No Compression | QF 80 | QF 40 | QF 10 |
|---|---|---|---|---|
| FGSM | 49.71 | 51.33 | 52.90 | 43.30 |
| ColorFool | 46.14 | 50.65 | 50.41 | 33.77 |
| FashionAdv | 18.40 | 19.90 | 20.49 | 20.82 |
Lower AP scores indicate better attack success.
# Clone the repository
git clone https://github.com/marc-treu/FashionAdv.git
cd FashionAdv
# Create conda environment
conda create -n fashionadv python=3.7
conda activate fashionadv# Install PyTorch (adjust CUDA version as needed)
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
# Install other requirements
pip install -r requirements.txt
# Install additional dependencies
pip install opencv-python pillow matplotlib tqdm
pip install pytorch-msssim kornia# Download MS-COCO validation set
wget http://images.cocodataset.org/zips/val2017.zip
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
# Extract to appropriate directories
unzip val2017.zip -d data/coco/
unzip annotations_trainval2017.zip -d data/coco/Our curated fashion style corpus contains 140 distinct clothing patterns selected from DeepFashion2 dataset.
# Download pre-processed fashion corpus
wget [LINK_TO_FASHION_CORPUS] -O fashion_corpus.zip
unzip fashion_corpus.zip -d Instance_Segmentation_Attack/data/We use SCHP (Self-Correction for Human Parsing) to generate clothing region masks.
# Download pre-computed masks for MS-COCO validation set
wget [LINK_TO_MASKS] -O clothing_masks.zip
unzip clothing_masks.zip -d Instance_Segmentation_Attack/data/mask_upper_shirt/import torch
from fashionAdv import fashionAdv_attack
# Load target image
content_index = 0 # Index in MS-COCO dataset
# Attack configuration
attack_setup = {
'max_iter': 200,
'lr': 0.02,
'adv_loss_weight': 1,
'textural_loss_weight': 200000,
'ssim_loss_weight': 50,
'tv_loss_weight': 0.00025,
}
# Generate adversarial example
adversarial_image = fashionAdv_attack(content_index, attack_setup)# Attack images 0-99 from the dataset
python fashionAdv.py 0 100
# Attack specific range
python fashionAdv.py 500 600FashionAdv consists of four main components:
- Clothing Segmentation: Uses SCHP to identify attackable regions
- Style Selection: Automatically selects optimal fashion style from corpus
- Adversarial Optimization: Generates robust adversarial textures
- Robustness Training: Uses EOT framework for transformation invariance
Our total loss combines adversarial and naturalness objectives:
L_total = Ξ±Β·L_adv + L_nat
L_nat = L_inc + Ξ²Β·L_tex
Where:
L_adv: Adversarial loss (classification + segmentation)L_inc: Inconspicuous loss (MS-SSIM + TV)L_tex: Texture transfer loss (content + style)
We simulate real-world conditions with:
- Perspective transformation: Random homography matrices
- Gaussian blur: Various kernel sizes and sigma values
- Color jittering: HSV adjustments
- Noise addition: Uniform noise injection
- JPEG compression: Quality factors 18-22
- Uncompressed: 18.40 AP (β26.16% vs. best baseline)
- JPEG QF 10: 20.82 AP (β33.14% vs. best baseline)
- SSIM: 0.958 (near-perfect structural similarity)
Key hyperparameters in attack_setup:
attack_setup = {
'max_iter': 200, # Optimization iterations
'lr': 0.02, # Learning rate
'adv_loss_weight': 1, # Adversarial loss weight
'textural_loss_weight': 200000, # Texture loss weight
'ssim_loss_weight': 50, # SSIM loss weight
'tv_loss_weight': 0.00025, # Total variation weight
# Robustness training probabilities
'apply_color_manipulation': 0.5,
'apply_transform': 0.5,
'apply_gaussian': 0.25,
'apply_jpeg': 0.75,
}Add your own fashion styles:
# Add new style image
new_style_path = 'path/to/your/style.jpg'
style_corpus.add_style(new_style_path)
# Recompute texture costs
compute_texture_costs(target_images, style_corpus)If you use FashionAdv in your research, please cite our paper:
@Inproceedings{marc-CVPRW2021,
Title = {Fashion-Guided Adversarial Attack on Person Segmentation},
Author = {Marc Treu and Trung-Nghia Le and Huy H. Nguyen and Junichi Yamagishi and Isao Echizen},
BookTitle = {Conference on Computer Vision and Pattern Recognition Workshops},
Year = {2021},
}- Marc Treu - Sorbonne University, France
- Trung-Nghia Le - National Institute of Informatics, Japan
- Huy H. Nguyen - National Institute of Informatics, Japan
- Junichi Yamagishi - NII, Japan & SOKENDAI, Japan
- Isao Echizen - NII, Japan & University of Tokyo, Japan
This project is licensed under the MIT License - see the LICENSE file for details.
- YOLACT: Real-time Instance Segmentation
- Self-Correction for Human Parsing
- Unrestricted Adversarial Examples
β Star this repository if you find it helpful!
π Found a bug? Please open an issue