Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution

IEEE Transactions on Image Processing, 2025

Zhangkai Ni¹, Yang Zhang¹, Wenhan Yang², Hanli Wang¹, Shiqi Wang³, Sam Kwong⁴

¹Tongji University, ²Peng Cheng Laboratory, ³City University of Hong Kong, ⁴Lingnan Univerity

This repository provides the official implementation for the paper "Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution," IEEE Transactions on Image Processing, vol. 34, pp. 3861-3872, Jun. 2025. Paper-official Paper-arXiv

About SSIU

Major efforts in data-driven image super-resolution (SR) primarily focus on expanding the receptive field of the model to better capture contextual information. However, these methods are typically implemented by stacking deeper networks or leveraging transformer-based attention mechanisms, which consequently increases model complexity. In contrast, model-driven methods based on the unfolding paradigm show promise in improving performance while effectively maintaining model compactness through sophisticated module design. Based on these insights, we propose a Structural Similarity-Inspired Unfolding (SSIU) method for efficient image SR. This method is designed through unfolding a SR optimization function constrained by structural similarity, aiming to combine the strengths of both data-driven and model-driven approaches. Our model operates progressively following the unfolding paradigm. Each iteration consists of multiple Mixed-Scale Gating Modules (MSGM) and an Efficient Sparse Attention Module (ESAM). The former implements comprehensive constraints on features, including a structural similarity constraint, while the latter aims to achieve sparse activation. In addition, we design a Mixture-of-Experts-based Feature Selector (MoE-FS) that fully utilizes multi-level feature information by combining features from different steps. Extensive experiments validate the efficacy and efficiency of our unfolding-inspired network. Our model outperforms current state-of-the-art models, boasting lower parameter counts and reduced memory consumption.

TL;DR: We propose a novel Structural Similarity-Inspired Unfolding (SSIU) network for lightweight image SR, which sparsely activates long-range pixels, leading to improved SR results with a lightweight architecture.

Experimental Results

Performance comparison of various lightweight SR models on five widely used benchmark datasets. All PSNR/SSIM values are computed on the Y-channel of the YCbCr color space. The FLOPs are measured corresponding to an SR image of size 1280 × 720 pixels. The top three performances are highlighted in red, orange, and yellow backgrounds, respectively.

Environment setup

To start, we prefer creating the environment using conda:

conda create -n ssiu
conda activate ssiu
pip install -r requirements.txt

Pytorch installation is machine dependent, please install the correct version for your machine.

Dependencies (click to expand)

• PyTorch, numpy: main computation.
• pytorch-msssim: SSIM calculation.
• tqdm: progress bar.
• opencv-python,scikit-image: image processing.
• imageio: images I/O.
• einops: torch tensor shaping with pretty api.
• PyYAML: parsing and generating YAML files.

Getting the data

The DF2K and benchmark are used in our paper and mainly includes the following datasets:

• Flickr2K
• DIV2K
• B100
• Manga109
• Set5
• Set14
• Urban100

Directory structure for the datasets

(click to expand;)

data_path
├── DF2K     # Link: https://www.kaggle.com/datasets/anvu1204/df2kdata?select=DF2K_train_HR
│   ├── DF2K_train_HR
│   └── DF2K_train_LR_bicubic
│        ├── X2
│        ├── X3
│        └── X4
└── benchmark     # Link: https://github.com/odefiao/SR-Test-Dataset
    ├── B100
    │   ├── HR
    │   └── LR_bicubic
    │       ├── X2
    │       ├── X3
    │       └── X4
    ├── Manga109
    │   ├── HR
    │   └── LR_bicubic
    │       ├── X2
    │       ├── X3
    │       └── X4
    ├── Set5
    │   ├── HR
    │   └── LR_bicubic
    │       ├── X2
    │       ├── X3
    │       └── X4
    ├── Set14
    │   ├── HR
    │   └── LR_bicubic
    │       ├── X2
    │       ├── X3
    │       └── X4
    └── Urban100
        ├── HR
        └── LR_bicubic
            ├── X2
            ├── X3
            └── X4

Running the model

Training

• Prepare the training dataset.
• Modify data_path in the ./configs/*.yml, which contains the DF2K and benchmark.
• Run the following commands for training:

# x2 commands
$ python train.py --config ./configs/SSUFSR_light_x2.yml
# x3 commands
$ python train.py --config ./configs/SSUFSR_light_x3.yml
# x4 commands
$ python train.py --config ./configs/SSUFSR_light_x4.yml

Testing

• Prepare the testing dataset.
• Modify data_path in the ./configs/*.yml, which contains the DF2K and benchmark.
• Prepare the pretrained model.
• Modify pretrain in the ./configs/*.yml, which corresponds to the path of the pretrained model.
• Run the following commands for tesing:

# x2 commands
$ python test.py --config ./configs/SSUFSR_light_x2.yml
# x3 commands
$ python test.py --config ./configs/SSUFSR_light_x3.yml
# x4 commands
$ python test.py --config ./configs/SSUFSR_light_x4.yml

Results

Pretrained models can be find in the ./pretrain_model folder.

Citation

If you find our work useful, please cite it as

@article{ni2025ssiu,
  title={Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution},
	author={Ni, Zhangkai, and Zhang, Yang, and Yang, Wenhan, and Wang, Hanli, and Wang, Shiqi and Kwong, Sam},
	journal={IEEE Transactions on Image Processing},
	volume={34},
	year={2025},
	publisher={IEEE}
}

Acknowledgments

This code is inspired by ELAN. We thank the authors for the nicely organized code!

Contact

Thanks for your attention! If you have any suggestion or question, feel free to leave a message here or contact Dr. Zhangkai Ni (eezkni@gmail.com).

License

MIT License