This repo is forked then modified from the initial repo for the course "Efficient Deep Learning" at IMT Atlantique | Brest | Spring 2025

The goal is to train a ResNet18 model in order to perform image classification on the dataset CIFAR10. Several fine-tuning and optimization method have been used:

1. Data Augmentation, Regularization
   • exploring hyper parameters on a fixed architecture
2. Quantization
   • quantification on a small convolutional network
3. Pruning
   • Pruning a small convolutional networks
4. Factorization

• Factorizing Deep neural networks

5. Distillation

• Distillation of knowledge and features between neural networks

6. Embedded Software and Hardware for Deep Learning
7. Final Session

• Challenge final results

Objectives:

Make sure you have download the CIFAR10 dataset at /opt/img/effdl-cifar10/ To run the code, make sure you are inside "~\efficient-deep-learning\LAB\cifar10_image_classification"

TLDR : this course is mostly based on a long project. The overarching goal is to explore the tradeoff between the performances of neural networks (= Accuracy on the test set) and complexity. Complexity can be either computational complexity (number of arithmetic operations), or memory complexity (memory size necessary to use the network).

We tested combinations of the various ideas mentioned above. We explored the tradeoff between architecture, number of parameters, and accuracy. Then, we studied new notions that open new avenues to explore this tradeoff : quantization, pruning, factorization, distillation. A deeper insight on how to thing about specific software or hardware architecture in order to fully exploit all the optimizations that can be done.

General References

List of references IMT Atlantique and AI

Amazon Book - Dive into Deep learning

Tutorial presentation on Efficient Deep Learning from NeurIPS'19

Training Deep Networks

Here are some academic papers discussing learning rate strategies :

• Cyclic learning rates
• Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Main strategies are readily available in pytorch.

Pytorch

Start page to access the full python API of pytorch, to check all existing functions.

A useful tutorial on Saving and Loading models.

Pytorch Cheat Sheet.

Data Augmentation

Popular methods :

Cut Out

Auto Augment

Other ressources :

A list of papers and code for data augmentation

IMGAUG and Colab Notebook showing how to use IMGAUG with pytorch

A popular python package in Kaggle competitions : Albumentations

Quantization

Binary Connect

XnorNet

BNN+

Whitepaper of quantization

Pruning

Pruning Filters for Efficient ConvNets

ThiNet

AutoML for Model Compression (AMC)

Pruning Channel with Attention Statistics (PCAS)

BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization

Factorization and operators

Deep Compression

Deep K-means

SqueezeNet

MobileNet

MobileNetV2

Shift Attention Layers

Distillation

Distilling the knowledge in a neural network

Fitnets: Hints for thin deep nets

LIT: Learned Intermediate Representation Training for Model Compression

A Comprehensive Overhaul of Feature Distillation

And the bit goes down: Revisiting the quantization of neural networks

Self-Supervised Learning

Pretext tasks used for learning Representations from data without labels

Embedded Software and Hardware

See references section of Tutorial presentation on Efficient Deep Learning from NeurIPS'19.

Companies / private sector

13 highest funded startups for hardware for DL

More complete list of companies working on hardware DL

Setting up on personal computer

Please see here for instructions on how to setup your environment on your personal computer.