知识蒸馏论文

Romero, A.; Ballas, N.; Kahou, S. E.; Chassang, A.; Gatta, C.; and Bengio, Y. 2015. Fitnets: Hints for thin deep nets.
In International Conference on Learning Representations (ICLR).

A Comprehensive Overhaul of Feature Distillation 特征图小于0的部分不进行强监督

A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning 提出FSP方法，用层与层的特征图之间的关系来蒸馏。

Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons 蒸馏对象是每层的切平面

PAYING MORE ATTENTION TO ATTENTION: IMPROVING THE PERFORMANCE OF CONVOLUTIONAL NEURAL NETWORKS VIA ATTENTION TRANSFER 通过Attention来蒸馏

Distilling the Knowledge in a Neural Network 蒸馏开山作品

Deep Mutual Learning 互相学习

Born Again Neural Networks 再生网络

FitNets: Hints for Thin Deep Nets 学生网络不仅仅拟合教师网络的soft-target，而且拟合隐藏层的输出（教师抽取的特征）；