Merge branch 'dev-1.x' into fix_nas

Author: twmht

.gitignore

@@ -11,7 +11,6 @@ __pycache__/
 .Python
 build/
 develop-eggs/
-dist/
 downloads/
 eggs/
 .eggs/

README.md

@@ -45,6 +45,20 @@
 
 English | [简体中文](README_zh-CN.md)
 
+<div align="center">
+  <a href="https://openmmlab.medium.com/" style="text-decoration:none;">
+    <img src="https://user-images.githubusercontent.com/25839884/218352562-cdded397-b0f3-4ca1-b8dd-a60df8dca75b.png" width="3%" alt="" /></a>
+  <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" />
+  <a href="https://discord.gg/raweFPmdzG" style="text-decoration:none;">
+    <img src="https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png" width="3%" alt="" /></a>
+  <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" />
+  <a href="https://twitter.com/OpenMMLab" style="text-decoration:none;">
+    <img src="https://user-images.githubusercontent.com/25839884/218346637-d30c8a0f-3eba-4699-8131-512fb06d46db.png" width="3%" alt="" /></a>
+  <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" />
+  <a href="https://www.youtube.com/openmmlab" style="text-decoration:none;">
+    <img src="https://user-images.githubusercontent.com/25839884/218346691-ceb2116a-465a-40af-8424-9f30d2348ca9.png" width="3%" alt="" /></a>
+</div>
+
 </div>
 
 ## Introduction

configs/_base_/settings/imagenet_bs2048_dmcp.py

@@ -0,0 +1,98 @@
+# dataset settings
+dataset_type = 'mmcls.ImageNet'
+
+max_search_epochs = 100
+# learning rate setting
+param_scheduler = [
+    # warm up learning rate scheduler
+    dict(
+        type='LinearLR',
+        start_factor=0.5,
+        by_epoch=True,
+        begin=0,
+        end=10,
+        convert_to_iter_based=True),
+    dict(
+        type='CosineAnnealingLR',
+        T_max=max_search_epochs,
+        eta_min=0.08,
+        by_epoch=True,
+        begin=10,
+        end=max_search_epochs,
+        convert_to_iter_based=True),
+]
+
+# optimizer setting
+paramwise_cfg = dict(norm_decay_mult=0.0, bias_decay_mult=0.0)
+
+optim_wrapper = dict(
+    constructor='mmrazor.SeparateOptimWrapperConstructor',
+    architecture=dict(
+        type='OptimWrapper',
+        optimizer=dict(type='SGD', lr=0.5, momentum=0.9, weight_decay=3e-4),
+        paramwise_cfg=paramwise_cfg),
+    mutator=dict(
+        type='OptimWrapper',
+        optimizer=dict(type='Adam', lr=0.5, weight_decay=1e-3)))
+
+# data preprocessor
+data_preprocessor = dict(
+    type='mmcls.ClsDataPreprocessor',
+    # RGB format normalization parameters
+    mean=[123.675, 116.28, 103.53],
+    std=[58.395, 57.12, 57.375],
+    # convert image from BGR to RGB
+    to_rgb=True,
+)
+train_pipeline = [
+    dict(type='LoadImageFromFile'),
+    dict(type='RandomResizedCrop', scale=224),
+    dict(type='ColorJitter', brightness=0.2, contrast=0.2, saturation=0.2),
+    dict(type='RandomFlip', prob=0.5, direction='horizontal'),
+    dict(type='PackClsInputs'),
+]
+
+test_pipeline = [
+    dict(type='LoadImageFromFile'),
+    dict(type='ResizeEdge', scale=256, edge='short'),
+    dict(type='CenterCrop', crop_size=224),
+    dict(type='PackClsInputs'),
+]
+
+train_dataloader = dict(
+    batch_size=64,
+    num_workers=4,
+    dataset=dict(
+        type=dataset_type,
+        data_root='data/imagenet',
+        ann_file='meta/train.txt',
+        data_prefix='train',
+        pipeline=train_pipeline),
+    sampler=dict(type='DefaultSampler', shuffle=True, _scope_='mmcls'),
+    persistent_workers=True,
+)
+
+val_dataloader = dict(
+    batch_size=64,
+    num_workers=4,
+    dataset=dict(
+        type=dataset_type,
+        data_root='data/imagenet',
+        ann_file='meta/val.txt',
+        data_prefix='val',
+        pipeline=test_pipeline),
+    sampler=dict(type='DefaultSampler', shuffle=True, _scope_='mmcls'),
+    persistent_workers=True,
+)
+val_evaluator = dict(type='mmcls.Accuracy', topk=(1, 5))
+
+# If you want standard test, please manually configure the test dataset
+test_dataloader = val_dataloader
+test_evaluator = val_evaluator
+
+evaluation = dict(interval=1, metric='accuracy')
+
+train_cfg = dict(by_epoch=True, max_epochs=max_search_epochs, val_interval=1)
+val_cfg = dict()
+test_cfg = dict()
+custom_hooks = [dict(type='DMCPSubnetHook')]

configs/distill/mmcls/dist/README.md

@@ -0,0 +1,45 @@
+# KD
+
+> [Knowledge Distillation from A Stronger Teacher](https://arxiv.org/abs/2205.10536)
+
+<!-- [ALGORITHM] -->
+
+## Abstract
+
+Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: this https URL .
+
+## Results and models
+
+### Classification
+
+| Location | Dataset  |      Teacher      |      Student      |  Acc  | Acc(T) | Acc(S) |       Config        | Download                                                         |
+| :------: | :------: | :---------------: | :---------------: | :---: | :----: | :----: | :-----------------: | :--------------------------------------------------------------- |
+|  logits  | ImageNet | [resnet34][r34_c] | [resnet18][r18_c] | 71.61 | 73.62  | 69.90  | [config][distill_c] | [teacher][r34_pth] \| [model][distill_pth] \| [log][distill_log] |
+
+**Note**
+
+There are fluctuations in the results of the experiments of DIST loss. For example, we run three times of the official code of DIST and get three different results.
+
+| Time | Top-1 |
+| ---- | ----- |
+| 1th  | 71.69 |
+| 2nd  | 71.82 |
+| 3rd  | 71.90 |
+
+## Citation
+
+```latex
+@article{huang2022knowledge,
+  title={Knowledge Distillation from A Stronger Teacher},
+  author={Huang, Tao and You, Shan and Wang, Fei and Qian, Chen and Xu, Chang},
+  journal={arXiv preprint arXiv:2205.10536},
+  year={2022}
+}
+```
+
+[distill_c]: ./dist_logits_resnet34_resnet18_8xb32_in1k.py
+[distill_log]: https://download.openmmlab.com/mmrazor/v1/distillation/dist_logits_resnet34_resnet18_8xb32_in1k.json
+[distill_pth]: https://download.openmmlab.com/mmrazor/v1/distillation/dist_logits_resnet34_resnet18_8xb32_in1k.pth
+[r18_c]: https://github.com/open-mmlab/mmclassification/blob/dev-1.x/configs/resnet/resnet18_8xb32_in1k.py
+[r34_c]: https://github.com/open-mmlab/mmclassification/blob/dev-1.x/configs/resnet/resnet34_8xb32_in1k.py
+[r34_pth]: https://download.openmmlab.com/mmclassification/v0/resnet/resnet34_8xb32_in1k_20210831-f257d4e6.pth

configs/distill/mmcls/dist/dist_logits_resnet34_resnet18_8xb32_in1k.py

@@ -0,0 +1,45 @@
+_base_ = [
+    'mmcls::_base_/datasets/imagenet_bs32.py',
+    'mmcls::_base_/schedules/imagenet_bs256.py',
+    'mmcls::_base_/default_runtime.py'
+]
+
+teacher_ckpt = 'https://download.openmmlab.com/mmclassification/v0/resnet/resnet34_8xb32_in1k_20210831-f257d4e6.pth'  # noqa: E501
+
+model = dict(
+    _scope_='mmrazor',
+    type='SingleTeacherDistill',
+    data_preprocessor=dict(
+        type='ImgDataPreprocessor',
+        # RGB format normalization parameters
+        mean=[123.675, 116.28, 103.53],
+        std=[58.395, 57.12, 57.375],
+        # convert image from BGR to RGB
+        bgr_to_rgb=True),
+    architecture=dict(
+        cfg_path='mmcls::resnet/resnet18_8xb32_in1k.py', pretrained=False),
+    teacher=dict(
+        cfg_path='mmcls::resnet/resnet34_8xb32_in1k.py', pretrained=False),
+    teacher_ckpt=teacher_ckpt,
+    distiller=dict(
+        type='ConfigurableDistiller',
+        student_recorders=dict(
+            fc=dict(type='ModuleOutputs', source='head.fc')),
+        teacher_recorders=dict(
+            fc=dict(type='ModuleOutputs', source='head.fc')),
+        distill_losses=dict(
+            loss_kl=dict(
+                type='DISTLoss',
+                inter_loss_weight=1.0,
+                intra_loss_weight=1.0,
+                tau=1,
+                loss_weight=2,
+            )),
+        loss_forward_mappings=dict(
+            loss_kl=dict(
+                logits_S=dict(from_student=True, recorder='fc'),
+                logits_T=dict(from_student=False, recorder='fc')))))
+
+val_cfg = dict(_delete_=True, type='mmrazor.SingleTeacherDistillValLoop')
+
+optim_wrapper = dict(optimizer=dict(nesterov=True))

configs/nas/mmcls/autoformer/AUTOFORMER_SUBNET_B.yaml

@@ -0,0 +1,134 @@
+backbone.base_embed_dims:
+  chosen: 64
+backbone.blocks.0.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.0.middle_channels:
+  chosen: 3.5
+backbone.blocks.0.mutable_mlp_ratios:
+  chosen: 3.5
+backbone.blocks.0.mutable_q_embed_dims:
+  chosen: 10
+backbone.blocks.1.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.1.middle_channels:
+  chosen: 3.5
+backbone.blocks.1.mutable_mlp_ratios:
+  chosen: 3.5
+backbone.blocks.1.mutable_q_embed_dims:
+  chosen: 64
+backbone.blocks.10.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.10.middle_channels:
+  chosen: 4.0
+backbone.blocks.10.mutable_mlp_ratios:
+  chosen: 4.0
+backbone.blocks.10.mutable_q_embed_dims:
+  chosen: 64
+backbone.blocks.11.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.11.middle_channels:
+  chosen: 576
+backbone.blocks.11.mutable_mlp_ratios:
+  chosen: 4.0
+backbone.blocks.11.mutable_q_embed_dims:
+  chosen: 10
+backbone.blocks.12.attn.mutable_attrs.num_heads:
+  chosen: 9
+backbone.blocks.12.middle_channels:
+  chosen: 4.0
+backbone.blocks.12.mutable_mlp_ratios:
+  chosen: 4.0
+backbone.blocks.12.mutable_q_embed_dims:
+  chosen: 9
+backbone.blocks.13.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.13.middle_channels:
+  chosen: 4.0
+backbone.blocks.13.mutable_mlp_ratios:
+  chosen: 4.0
+backbone.blocks.13.mutable_q_embed_dims:
+  chosen: 10
+backbone.blocks.14.attn.mutable_attrs.num_heads:
+  chosen: 8
+backbone.blocks.14.middle_channels:
+  chosen: 576
+backbone.blocks.14.mutable_mlp_ratios:
+  chosen: 3.5
+backbone.blocks.14.mutable_q_embed_dims:
+  chosen: 8
+backbone.blocks.15.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.15.middle_channels:
+  chosen: 3.0
+backbone.blocks.15.mutable_mlp_ratios:
+  chosen: 3.0
+backbone.blocks.15.mutable_q_embed_dims:
+  chosen: 10
+backbone.blocks.2.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.2.middle_channels:
+  chosen: 576
+backbone.blocks.2.mutable_mlp_ratios:
+  chosen: 3.5
+backbone.blocks.2.mutable_q_embed_dims:
+  chosen: 10
+backbone.blocks.3.attn.mutable_attrs.num_heads:
+  chosen: 8
+backbone.blocks.3.middle_channels:
+  chosen: 4.0
+backbone.blocks.3.mutable_mlp_ratios:
+  chosen: 4.0
+backbone.blocks.3.mutable_q_embed_dims:
+  chosen: 8
+backbone.blocks.4.attn.mutable_attrs.num_heads:
+  chosen: 10
+backbone.blocks.4.middle_channels:
+  chosen: 576
+backbone.blocks.4.mutable_mlp_ratios:
+  chosen: 3.0
+backbone.blocks.4.mutable_q_embed_dims:
+  chosen: 10
+backbone.blocks.5.attn.mutable_attrs.num_heads:
+  chosen: 9
+backbone.blocks.5.middle_channels:
+  chosen: 3.0
+backbone.blocks.5.mutable_mlp_ratios:
+  chosen: 3.0
+backbone.blocks.5.mutable_q_embed_dims:
+  chosen: 9
+backbone.blocks.6.attn.mutable_attrs.num_heads:
+  chosen: 8
+backbone.blocks.6.middle_channels:
+  chosen: 576
+backbone.blocks.6.mutable_mlp_ratios:
+  chosen: 3.5
+backbone.blocks.6.mutable_q_embed_dims:
+  chosen: 8
+backbone.blocks.7.attn.mutable_attrs.num_heads:
+  chosen: 8
+backbone.blocks.7.middle_channels:
+  chosen: 3.5
+backbone.blocks.7.mutable_mlp_ratios:
+  chosen: 3.5
+backbone.blocks.7.mutable_q_embed_dims:
+  chosen: 8
+backbone.blocks.8.attn.mutable_attrs.num_heads:
+  chosen: 9
+backbone.blocks.8.middle_channels:
+  chosen: 576
+backbone.blocks.8.mutable_mlp_ratios:
+  chosen: 4.0
+backbone.blocks.8.mutable_q_embed_dims:
+  chosen: 9
+backbone.blocks.9.attn.mutable_attrs.num_heads:
+  chosen: 8
+backbone.blocks.9.middle_channels:
+  chosen: 576
+backbone.blocks.9.mutable_mlp_ratios:
+  chosen: 4.0
+backbone.blocks.9.mutable_q_embed_dims:
+  chosen: 8
+backbone.mutable_depth:
+  chosen: 14
+backbone.mutable_embed_dims:
+  chosen: 576

configs/nas/mmcls/autoformer/README.md

@@ -44,15 +44,16 @@ CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 ./tools/dist_train.sh \
 ```bash
 CUDA_VISIBLE_DEVICES=0 PORT=29500 ./tools/dist_test.sh \
   configs/nas/mmcls/autoformer/autoformer_subnet_8xb128_in1k.py \
-  $STEP2_CKPT 1 --work-dir $WORK_DIR \
-  --cfg-options algorithm.mutable_cfg=$STEP2_SUBNET_YAML
+  none 1 --work-dir $WORK_DIR \
+  --cfg-options model.init_cfg.checkpoint=$STEP1_CKPT model.init_weight_from_supernet=True
+
 ```
 
 ## Results and models
 
-| Dataset  | Supernet |                                                                                                                                               Subnet                                                                                                                                                | Params(M) | Flops(G) | Top-1 (%) | Top-5 (%) |                     Config                      | Download                                                                                                                                                                    |     Remarks      |
-| :------: | :------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-------: | :------: | :-------: | :-------: | :---------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------: |
-| ImageNet |   vit    | [mutable](https://openmmlab-share.oss-cn-hangzhou.aliyuncs.com/mmrazor/v0.1/nas/spos/spos_shufflenetv2_subnet_8xb128_in1k/spos_shufflenetv2_subnet_8xb128_in1k_flops_0.33M_acc_73.87_20211222-454627be_mutable_cfg.yaml?versionId=CAEQHxiBgICw5b6I7xciIGY5MjVmNWFhY2U5MjQzN2M4NDViYzI2YWRmYWE1YzQx) |  52.472   |   10.2   |   82.48   |   95.99   | [config](./autoformer_supernet_32xb256_in1k.py) | [model](https://openmmlab-share.oss-cn-hangzhou.aliyuncs.com/mmrazor/x.pth) \| [log](https://openmmlab-share.oss-cn-hangzhou.aliyuncs.com/mmrazor/v0.1/nas/spos/x.log.json) | MMRazor searched |
+| Dataset  | Supernet |                               Subnet                               | Params(M) | Flops(G) | Top-1 (%) | Top-5 (%) |                     Config                      | Download                                                                                                                                                                                                                                                  |     Remarks      |
+| :------: | :------: | :----------------------------------------------------------------: | :-------: | :------: | :-------: | :-------: | :---------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------: |
+| ImageNet |   vit    | [mutable](./configs/nas/mmcls/autoformer/AUTOFORMER_SUBNET_B.yaml) |  54.319   |  10.57   |   82.47   |   95.99   | [config](./autoformer_supernet_32xb256_in1k.py) | [model](https://download.openmmlab.com/mmrazor/v1/autoformer/autoformer_supernet_32xb256_in1k_20220919_110144-c658ce8f.pth) \| [log](https://download.openmmlab.com/mmrazor/v1/autoformer/autoformer_supernet_32xb256_in1k_20220919_110144-c658ce8f.json) | MMRazor searched |
 
 **Note**:
 

configs/nas/mmcls/autoformer/autoformer_subnet_8xb256_in1k.py

@@ -0,0 +1,17 @@
+_base_ = 'autoformer_supernet_32xb256_in1k.py'
+
+model = dict(
+    _scope_='mmrazor',
+    type='sub_model',
+    cfg=_base_.supernet,
+    # NOTE: You can replace the yaml with the mutable_cfg searched by yourself
+    fix_subnet='configs/nas/mmcls/autoformer/AUTOFORMER_SUBNET_B.yaml',
+    # You can also load the checkpoint of supernet instead of the specific
+    # subnet by modifying the `checkpoint`(path) in the following `init_cfg`
+    # with `init_weight_from_supernet = True`.
+    init_weight_from_supernet=False,
+    init_cfg=dict(
+        type='Pretrained',
+        checkpoint=  # noqa: E251
+        'https://download.openmmlab.com/mmrazor/v1/autoformer/autoformer_supernet_32xb256_in1k_20220919_110144-c658ce8f.pth',  # noqa: E501
+        prefix='architecture.'))