Constraint-Relaxation-Attack

Official implementation for the paper: Efficient Robustness Evaluation via Constraint Relaxation

Overview

This repository contains the implementation of the Constraint Relaxation Attack (CRAttack), a novel approach for efficiently evaluating the adversarial robustness of deep neural networks. The attack relaxes constraints during the optimization process to find more effective adversarial examples.

Environment Setup

• OS: Ubuntu 20.04.3
• GPU: NVIDIA Tesla V100
• CUDA: 11.4
• Python: 3.8.10
• PyTorch: 1.10.1
• Torchvision: 0.11.2

Installation

git clone https://github.com/fzjcdt/constraint-relaxation-attack.git
cd constraint-relaxation-attack
pip install -r requirements.txt

Quick Start Example

The following example demonstrates how to use CRAttack on a pre-trained model from RobustBench against the CIFAR-10 dataset:

import torch
import torchvision.transforms as transforms
from robustbench import load_model
from torch.utils.data import DataLoader
from torchvision.datasets import CIFAR10

from attacks import CRAttack

# Set device for computation
device = 'cuda:0' if torch.cuda.is_available() else 'cpu'

# Load CIFAR-10 test dataset
test_loader = DataLoader(CIFAR10('./data/cifar10', train=False, transform=transforms.ToTensor()),
                         batch_size=10000, shuffle=False, num_workers=0)

# Extract data from loader
X, y = [], []
for i, (x, y_) in enumerate(test_loader):
    X = x.to(device)
    y = y_.to(device)

# Load pre-trained robust model from RobustBench
model = load_model(model_name='Wang2023Better_WRN-28-10', dataset='cifar10', threat_model='Linf').to(device)
model = model.eval()

# Initialize the CRAttack with epsilon=8/255 and specify log file
attacker = CRAttack(model, eps=8.0 / 255, log_path='cr_test.log')

# Run evaluation with batch size of 200
attacker.run_standard_evaluation(X, y, bs=200)

This example:

1. Sets up the computation device (GPU if available)
2. Loads the CIFAR-10 test dataset
3. Loads a pre-trained WideResNet model from RobustBench
4. Creates a CRAttack instance with perturbation bound ε=8/255
5. Runs the attack evaluation and logs results to cr_test.log

Attack Commands

You can run attacks on various datasets using our provided scripts. The default setting evaluates all models specified in the ./model_ids/ directory for the given dataset.

# Attack models on CIFAR-10 with default epsilon (8/255)
python main.py --dataset 'cifar10'

# Attack models on CIFAR-100 with default epsilon (8/255)
python main.py --dataset 'cifar100'

# Attack models on ImageNet with epsilon=4/255
python main.py --dataset 'imagenet' --eps '4/255'

Custom Model Evaluation

If you want to evaluate just one specific model, use the --model_id parameter:

python main.py --dataset 'cifar10' --model_id 'Wang2023Better_WRN-28-10'

Full results

CIFAR-10

Linf, eps=8/255

#	Model ID (Paper)	Architecture	Best known robust accuracy	AutoAttack robust accuracy	AutoAttack forward number	AutoAttack backward number	CR attack robust accuracy	CR attack forward number	CR attack backward number
1	Bartoldson2024Adversarial_WRN-94-16	WideResNet-94-16	73.71%	73.71%	5810	1443	73.59%	108 (53.8×)	65 (22.0×)
2	Amini2024MeanSparse	MeanSparse RaWideResNet-70-16	72.08%	72.08%	5674	1399	71.85%	104 (54.6×)	62 (22.6×)
3	Bartoldson2024Adversarial_WRN-82-8	WideResNet-82-8	71.59%	71.59%	5647	1393	71.42%	107 (52.8×)	64 (21.8×)
4	Peng2023Robust	RaWideResNet-70-16	71.07%	71.07%	5637	1390	70.99%	104 (54.2×)	63 (22.1×)
5	Wang2023Better_WRN-70-16	WideResNet-70-16	70.69%	70.69%	5541	1370	70.56%	119 (46.6×)	73 (18.8×)
6	Cui2023Decoupled_WRN-28-10	WideResNet-28-10	67.73%	67.73%	5344	1322	67.55%	104 (51.4×)	62 (21.3×)
7	Bai2023Improving_edm	ResNet-152 + WideResNet-70-16 + mixing network	68.06%	68.06%	5459	1354	67.35%	145 (37.7×)	92 (14.7×)
8	Wang2023Better_WRN-28-10	WideResNet-28-10	67.31%	67.31%	5338	1322	67.21%	118 (45.2×)	66 (18.4×)
9	Rebuffi2021Fixing_70_16_cutmix_extra	WideResNet-70-16	66.56%	66.58%	4998	1243	66.51%	111 (45.0×)	67 (18.6×)
10	Gowal2021Improving_70_16_ddpm_100m	WideResNet-70-16	66.10%	66.11%	5148	1275	66.08%	109 (47.2×)	66 (19.3×)
11	Gowal2020Uncovering_70_16_extra	WideResNet-70-16	65.87%	65.88%	5055	1253	65.74%	117 (43.2×)	71 (17.6×)
12	Huang2022Revisiting_WRN-A4	WideResNet-A4	65.79%	65.79%	5210	1289	65.71%	113 (46.1×)	69 (18.7×)
13	Rebuffi2021Fixing_106_16_cutmix_ddpm	WideResNet-106-16	64.58%	64.64%	4977	1234	64.47%	116 (42.9×)	71 (17.4×)
14	Rebuffi2021Fixing_70_16_cutmix_ddpm	WideResNet-70-16	64.20%	64.25%	4915	1220	64.18%	116 (42.4×)	71 (17.2×)
15	Gowal2021Improving_28_10_ddpm_100m	WideResNet-28-10	63.38%	63.44%	4928	1221	63.36%	108 (45.6×)	65 (18.8×)
16	Pang2022Robustness_WRN70_16	WideResNet-70-16	63.35%	63.35%	4868	1209	63.28%	115 (42.3×)	70 (17.3×)
17	Rade2021Helper_extra	WideResNet-34-10	62.83%	62.83%	4793	1193	62.66%	111 (43.2×)	68 (17.5×)
18	Sehwag2021Proxy_ResNest152	ResNest152	62.79%	62.79%	4758	1181	62.53%	110 (43.3×)	66 (17.9×)
19	Gowal2020Uncovering_28_10_extra	WideResNet-28-10	62.76%	62.80%	4729	1176	62.71%	121 (39.1×)	74 (15.9×)
20	Huang2021Exploring_ema	WideResNet-34-R	62.50%	62.54%	4824	1200	62.49%	106 (45.5×)	64 (18.8×)
21	Huang2021Exploring	WideResNet-34-R	61.56%	61.56%	4665	1160	61.59%	102 (45.7×)	61 (19.0×)
22	Dai2021Parameterizing	WideResNet-28-10-PSSiLU	61.55%	61.55%	4776	1188	61.45%	108 (44.2×)	66 (18.0×)
23	Pang2022Robustness_WRN28_10	WideResNet-28-10	61.04%	61.04%	4774	1186	60.91%	118 (40.5×)	72 (16.5×)
24	Rade2021Helper_ddpm	WideResNet-28-10	60.97%	60.97%	4727	1174	60.80%	109 (43.4×)	66 (17.8×)
25	Rebuffi2021Fixing_28_10_cutmix_ddpm	WideResNet-28-10	60.73%	60.75%	4728	1175	60.68%	117 (40.4×)	72 (16.3×)
26	Sridhar2021Robust_34_15	WideResNet-34-15	60.41%	60.41%	4664	1159	60.32%	132 (35.3×)	82 (14.1×)
27	Sehwag2021Proxy	WideResNet-34-10	60.27%	60.27%	4648	1155	60.23%	104 (44.7×)	62 (18.6×)
28	Wu2020Adversarial_extra	WideResNet-28-10	60.04%	60.04%	4667	1162	59.98%	120 (38.9×)	74 (15.7×)
29	Sridhar2021Robust	WideResNet-28-10	59.66%	59.66%	4668	1163	59.57%	109 (42.8×)	66 (17.6×)
30	Zhang2020Geometry	WideResNet-28-10	59.64%	59.64%	4641	1160	59.12%	158 (29.4×)	100 (11.6×)
31	Carmon2019Unlabeled	WideResNet-28-10	59.53%	59.53%	4558	1137	59.46%	109 (41.8×)	66 (17.2×)
32	Gowal2021Improving_R18_ddpm_100m	PreActResNet-18	58.5%	58.63%	4557	1136	58.60%	101 (45.1×)	61 (18.6×)
33	Addepalli2021Towards_WRN34	WideResNet-34-10	58.04%	58.04%	4452	1113	58.00%	135 (33.0×)	85 (13.1×)
34	Addepalli2022Efficient_WRN_34_10	WideResNet-34-10	57.81%	57.81%	4558	1136	57.72%	119 (38.3×)	74 (15.4×)
35	Chen2021LTD_WRN34_20	WideResNet-34-20	57.71%	57.71%	4572	1139	57.68%	132 (34.6×)	83 (13.7×)
36	Rade2021Helper_R18_extra	PreActResNet-18	57.67%	57.67%	4573	1140	57.49%	113 (40.5×)	69 (16.5×)
37	Jia2022LAS-AT_70_16	WideResNet-70-16	57.61%	57.61%	4461	1110	57.56%	114 (39.1×)	70 (15.9×)
38	Debenedetti2022Light_XCiT-L12	XCiT-L12	57.58%	57.58%	4408	1105	57.63%	94 (46.9×)	56 (19.7×)
39	Debenedetti2022Light_XCiT-M12	XCiT-M12	57.27%	57.27%	4531	1133	57.28%	100 (45.3×)	60 (18.9×)
40	Sehwag2020Hydra	WideResNet-28-10	57.14%	57.14%	4495	1120	57.12%	106 (42.4×)	64 (17.5×)
41	Gowal2020Uncovering_70_16	WideResNet-70-16	57.14%	57.2%	4431	1105	57.10%	106 (41.8×)	64 (17.3×)
42	Rade2021Helper_R18_ddpm	PreActResNet-18	57.09%	57.09%	4431	1104	57.01%	111 (39.9×)	68 (16.2×)
43	Chen2021LTD_WRN34_10	WideResNet-34-10	56.94%	56.94%	4386	1095	56.87%	125 (35.1×)	78 (14.0×)
44	Gowal2020Uncovering_34_20	WideResNet-34-20	56.82%	56.86%	4246	1062	56.74%	112 (37.9×)	68 (15.6×)
45	Rebuffi2021Fixing_R18_ddpm	PreActResNet-18	56.66%	56.66%	4272	1064	56.57%	113 (37.8×)	69 (15.4×)
46	Wang2020Improving	WideResNet-28-10	56.29%	56.29%	4482	1119	56.43%	119 (37.7×)	73 (15.3×)
47	Jia2022LAS-AT_34_10	WideResNet-34-10	56.26%	56.26%	4335	1080	56.19%	116 (37.4×)	71 (15.2×)
48	Wu2020Adversarial	WideResNet-34-10	56.17%	56.17%	4336	1080	56.07%	109 (39.8×)	66 (16.4×)
49	Debenedetti2022Light_XCiT-S12	XCiT-S12	56.14%	56.14%	4296	1076	55.98%	108 (39.8×)	65 (16.6×)
50	Sehwag2021Proxy_R18	ResNet-18	55.54%	55.54%	4244	1060	55.66%	104 (40.8×)	63 (16.8×)
51	Hendrycks2019Using	WideResNet-28-10	54.92%	54.92%	4323	1079	54.88%	104 (41.6×)	63 (17.1×)
52	Pang2020Boosting	WideResNet-34-20	53.74%	53.74%	4200	1062	53.72%	301 (14.0×)	203 (5.2×)
53	Cui2020Learnable_34_20	WideResNet-34-20	53.57%	53.57%	4141	1039	53.11%	90 (46.0×)	53 (19.6×)
54	Zhang2020Attacks	WideResNet-34-10	53.51%	53.51%	4136	1035	53.44%	118 (35.1×)	73 (14.2×)
55	Rice2020Overfitting	WideResNet-34-20	53.42%	53.42%	4138	1037	53.43%	106 (39.0×)	64 (16.2×)
56	Huang2020Self	WideResNet-34-10	53.34%	53.34%	4072	1018	52.83%	111 (36.7×)	68 (15.0×)
57	Zhang2019Theoretically	WideResNet-34-10	53.08%	53.08%	4089	1024	52.45%	97 (42.2×)	58 (17.7×)
58	Cui2020Learnable_34_10	WideResNet-34-10	52.86%	52.86%	4014	1008	52.33%	84 (47.8×)	49 (20.6×)
59	Addepalli2022Efficient_RN18	ResNet-18	52.48%	52.48%	3968	997	52.45%	125 (31.7×)	78 (12.8×)
60	Chen2020Adversarial	ResNet-50 (3x ensemble)	51.56%	51.56%	4090	1025	51.50%	188 (21.8×)	123 (8.3×)

CIFAR-100

Linf, eps=8/255

#	Model ID (Paper)	Architecture	Best known robust accuracy	AutoAttack robust accuracy	AutoAttack forward number	AutoAttack backward number	CR attack robust accuracy	CR attack forward number	CR attack backward number
1	Wang2023Better_WRN-70-16	WideResNet-70-16	42.67%	42.67%	3351	844	42.57%	119 (28.2×)	71 (11.9×)
2	Cui2023Decoupled_WRN-28-10	WideResNet-28-10	39.18 %	39.18 %	3009	764	39.13%	96 (31.3 ×)	54 (14.1×)
3	Wang2023Better_WRN-28-10	WideResNet-28-10	38.83%	38.83%	2959	749	38.67%	116 (25.5×)	69 (10.9×)
4	Bai2023Improving_edm	ResNet-152+WideResNet-70-16+mixing-network	38.72 %	38.72 %	3082	789	38.63%	94 (32.8×)	54 (14.6×)
5	Gowal2020Uncovering_extra	WideResNet-70-16	36.88%	36.88%	2695	686	36.87%	95 (28.4×)	54 (12.7×)
6	Debenedetti2022Light_XCiT-L12	XCiT-L12	35.08%	35.08%	2665	680	34.96%	99 (26.9×)	57 (11.9×)
7	Bai2023Improving_trades	ResNet-152+WideResNet-70-16+mixing-network	35.15 %	35.15 %	2679	693	34.80%	85 (31.5×)	47 (14.7×)
8	Rebuffi2021Fixing_70_16_cutmix_ddpm	WideResNet-70-16	34.64%	34.64%	2594	658	34.56%	99 (26.2×)	57 (11.5×)
9	Debenedetti2022Light_XCiT-M12	XCiT-M12	34.21%	34.21%	2724	692	34.09%	103 (26.4×)	61 (11.3×)
10	Pang2022Robustness_WRN70_16	WideResNet-70-16	33.05%	33.05%	2595	659	32.97%	96 (27.0×)	55 (12.0×)
11	Cui2023Decoupled_WRN-34-10_autoaug	WideResNet-34-10	32.52 %	32.52 %	2597	661	32.46%	81 (32.1×)	45 (14.7×)
12	Debenedetti2022Light_XCiT-S12	XCiT-S12	32.19%	32.19%	2522	645	32.10%	104 (24.2×)	61 (10.6×)
13	Rebuffi2021Fixing_28_10_cutmix_ddpm	WideResNet-28-10	32.06%	32.06%	2470	628	31.95%	95 (26.0×)	54 (11.6×)
14	Jia2022LAS-AT_34_20	WideResNet-34-20	31.91%	31.91%	2554	653	31.92%	88 (29.0×)	49 (13.3×)
15	Addepalli2022Efficient_WRN_34_10	WideResNet-34-10	31.85%	31.85%	2363	604	31.81%	98 (24.1×)	56 (10.8×)
16	Cui2023Decoupled_WRN-34-10	WideResNet-34-10	31.65 %	31.65 %	2518	640	31.62%	82 (30.7×)	45 (14.2×)
17	Sehwag2021Proxy	WideResNet-34-10	31.15%	31.15%	2506	641	31.14%	96 (26.1×)	55 (11.7×)
18	Chen2024Data_WRN_34_10	WideResNet-34-10	31.13 %	31.13 %	2456	626	31.12%	85 (28.9×)	47 (13.3×)
19	Cui2020Learnable_34_10_LBGAT9_eps_8_255	WideResNet-34-10	31.20%	31.20%	2604	667	31.10%	113 (23.0×)	67 (10.0×)
20	Pang2022Robustness_WRN28_10	WideResNet-28-10	31.08%	31.08%	2518	640	31.03%	93 (27.1×)	53 (12.1×)
21	Jia2022LAS-AT_34_10	WideResNet-34-10	30.77%	30.77%	2469	627	30.77%	86 (28.7×)	48 (13.1×)
22	Chen2021LTD_WRN34_10	WideResNet-34-10	30.59%	30.59%	2333	598	30.58%	96 (24.3×)	55 (10.9×)
23	Addepalli2021Towards_WRN34	WideResNet-34-10	30.35%	30.35%	2566	653	30.23%	94 (27.3×)	54 (12.1×)
24	Cui2020Learnable_34_20_LBGAT6	WideResNet-34-20	30.20%	30.20%	2379	609	29.87%	93 (25.6×)	53 (11.5×)
25	Gowal2020Uncovering	WideResNet-70-16	30.03%	30.03%	2389	607	29.99%	85 (28.1×)	48 (12.6×)
26	Cui2020Learnable_34_10_LBGAT6	WideResNet-34-10	29.33%	29.33%	2377	607	28.87%	102 (23.3×)	60 (10.1×)
27	Rade2021Helper_R18_ddpm	PreActResNet-18	28.88%	28.88%	2235	571	28.80%	85 (26.3×)	48 (11.9×)
28	Wu2020Adversarial	WideResNet-34-10	28.86%	28.86%	2345	598	28.85%	84 (27.9×)	47 (12.7×)
29	Rebuffi2021Fixing_R18_ddpm	PreActResNet-18	28.50%	28.50%	2259	572	28.37%	89 (25.4×)	50 (11.4×)
30	Hendrycks2019Using	WideResNet-28-10	28.42%	28.42%	2256	580	28.47%	91 (24.8×)	52 (11.2×)

ImageNet

Linf, eps=4/255

#	Model ID (Paper)	Architecture	Best known robust accuracy	AutoAttack robust accuracy	AutoAttack forward number	AutoAttack backward number	CR attack robust accuracy	CR attack forward number	CR attack backward number
1	Amini2024MeanSparse	MeanSparse ConvNeXt-L	59.64%	59.64%	5075	1248	59.70	165(30.8×)	99(12.6×)
2	Liu2023Comprehensive_Swin-L	Swin-L	59.56%	59.56%	4918	1212	59.46%	165(29.8×)	99(12.2×)
3	Liu2023Comprehensive_ConvNeXt-L	ConvNeXt-L	58.48%	58.48%	5013	1235	58.50%	161(31.1×)	96(12.9×)
4	Singh2023Revisiting_ConvNeXt-L-ConvStem	ConvNeXt-L+ConvStem	57.7%	57.7%	5106	1257	57.62%	158(32.3×)	94(13.4×)
5	Liu2023Comprehensive_Swin-B	Swin-B	56.16%	56.16%	4795	1183	56.1%	156(30.7×)	93(12.7×)
6	Singh2023Revisiting_ConvNeXt-B-ConvStem	ConvNeXt-B+ConvStem	56.14%	56.14%	4946	1215	56.04%	157(31.5×)	93(13.1×)
7	Liu2023Comprehensive_ConvNeXt-B	ConvNeXt-B	55.82%	55.82%	4699	1159	55.8%	153(30.7×)	91(12.7×)
8	Singh2023Revisiting_ViT-B-ConvStem	ViT-B+ConvStem	54.66%	54.66%	4638	1145	54.6%	154(30.1×)	92(12.4×)
9	Singh2023Revisiting_ConvNeXt-S-ConvStem	ConvNeXt-S+ConvStem	52.42%	52.42%	4514	1116	52.28%	148(30.5×)	88(12.7×)
10	Singh2023Revisiting_ConvNeXt-T-ConvStem	ConvNeXt-T+ConvStem	49.46%	49.46%	4416	1093	49.46%	143(30.9×)	84(13.0×)
11	Peng2023Robust	RaWideResNet-101-2	48.94%	48.94%	4140	1028	48.84%	140(29.6×)	82(12.5×)
12	Singh2023Revisiting_ViT-S-ConvStem	ViT-S+ConvStem	48.08%	48.08%	4198	1038	48.04%	142(29.6×)	84(12.4×)
13	Debenedetti2022Light_XCiT-L12	XCiT-L12	47.60%	47.60%	3863	964	47.52%	110 (35.1×)	64 (15.1×)
14	Debenedetti2022Light_XCiT-M12	XCiT-M12	45.24%	45.24%	3751	935	45.20%	110 (34.1×)	65 (14.4×)
15	Debenedetti2022Light_XCiT-S12	XCiT-S12	41.78%	41.78%	3464	874	41.64%	105 (33.0×)	61 (14.3×)