Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space

1 code implementation • NeurIPS 2020 • Shangchen Du, Shan You, Xiaojie Li, Jianlong Wu, Fei Wang, Chen Qian, ChangShui Zhang

In this paper, we examine the diversity of teacher models in the gradient space and regard the ensemble knowledge distillation as a multi-objective optimization problem so that we can determine a better optimization direction for the training of student network.

Knowledge Distillation