Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons

Official Pytorch implementation of paper:

Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons (AAAI 2019).

Slides and poster are available on homepage

Environment

Python 3.6, Pytorch 0.4.1, Torchvision

Knowledge distillation (CIFAR-10)

cifar10_AB_distillation.py

Distillation from WRN 22-4 (teacher) to WRN 16-2 (student) on CIFAR-10 dataset.

Pre-trained teacher network (WRN 22-4) is included. Just run the code.

Transfer learning (MIT_scenes)

MITscenes_AB_distillation.py

Transfer learning from ImageNet pre-trained model (teacher) to randomly initialized model (student).

Teacher : ImageNet pre-trained ResNet 50

Student : MobileNet or MobileNetV2 (randomly initialized model)

Please change base learning rate to 0.1 for MobileNetV2.

MIT_scenes dataset should be arranged for Torchvision ImageFolder function.

Train set : $dataset_path / train / $class_name / $image_name

Test set : $dataset_path / test / $class_name / $image name

and run with dataset path.

MobileNet

python MITscenes_AB_distillation.py --data_root $dataset_path

MobileNet V2

python MITscenes_AB_distillation.py --data_root $dataset_path --network mobilenetV2

Other implementations

Tensorflow: https://github.com/sseung0703/Knowledge_distillation_methods_wtih_Tensorflow

Citation

@inproceedings{ABdistill,
	title = {Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons},
	author = {Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi},
	booktitle = {AAAI Conference on Artificial Intelligence (AAAI)},
	year = {2019}
}