External Knowledge Injection for CLIP-Based Class-Incremental Learning (ICCV 2025)

Da-Wei Zhou  Kai-Wen Li  Jingyi Ning  Han-Jia Ye  Lijun Zhang  De-Chuan Zhan 

State Key Laboratory for Novel Software Technology, Nanjing University

The code repository for "External Knowledge Injection for CLIP-Based Class-Incremental Learning" (ICCV 2025) in PyTorch. If you use any content of this repo for your work, please cite the following bib entry:

@article{zhou2025external,
  title={External Knowledge Injection for CLIP-Based Class-Incremental Learning},
  author={Da-Wei Zhou and Kai-Wen Li and Jingyi Ning and Han-Jia Ye and Lijun Zhang and De-Chuan Zhan},
  journal={arXiv preprint arXiv:2503.08510},
  year={2025}
}

📢 Updates

[06/2025] Accepted to ICCV 2025.

[03/2025] Code has been released.

[03/2025] arXiv paper has been released.

📝 Introduction

Class-Incremental Learning (CIL) enables learning systems to continuously adapt to evolving data streams. With the advancement of pre-training, leveraging pre-trained vision-language models (eg., CLIP) offers a promising starting point for CIL. However, CLIP makes decisions by matching visual embeddings to class names, overlooking the rich contextual information conveyed through language. For instance, the concept of “cat” can be decomposed into features like tail, fur, and face for recognition. Besides, since the model is continually updated, these detailed features are overwritten in CIL, requiring external knowledge for compensation. In this paper, we introduce ExterNal knowledGe INjEction (ENGINE) for CLIP-based CIL. To enhance knowledge transfer from outside the dataset, we propose a dual-branch injection tuning framework that encodes informative knowledge from both visual and textual modalities. The visual branch is enhanced with data augmentation to enrich the visual features, while the textual branch leverages GPT-4 to rewrite discriminative descriptors. In addition to this on-the-fly knowledge injection, we also implement post-tuning knowledge by re-ranking the prediction results during inference. With the injected knowledge, the model can better capture informative features for downstream tasks as data evolves. Extensive experiments demonstrate the state-of-the-art performance of ENGINE.

🔧 Requirements

Environment

1 torch 1.13.0

2 torchvision 0.15.0

3 open-clip 2.30.0

Dataset

We provide the processed datasets as follows:

• CIFAR100: will be automatically downloaded by the code.
• CUB200: Google Drive: link or OneDrive link
• ImageNet-R: Google Drive: link or Onedrive: link
• ObjectNet: Onedrive: link You can also refer to the filelist and processing code if the file is too large to download.
• Cars: Google Drive: link or OneDrive: link
• UCF: Google Drive: link or OneDrive: link
• Aircraft: Google Drive: link or OneDrive: link
• Food: Google Drive: link or OneDrive: link
• SUN: OneDrive: link

These subsets are sampled from the original datasets. Please note that I do not have the right to distribute these datasets. If the distribution violates the license, I shall provide the filenames instead.

You need to modify the path of the datasets in ./utils/data.py according to your own path.

💡 Running scripts

To prepare your JSON files, refer to the settings in the exps folder and run the following command. All main experiments from the paper are already provided in the exps folder, you can simply execute them to reproduce the results found in the logs folder.

python main.py --config ./exps/engine/[configname].json

🎈 Acknowledgement

This repo is based on CIL_Survey and PyCIL.

💭 Correspondence

If you have any questions, please contact me via email or open an issue.