.png){kind=icon}
Return
Knowledge Distillation With Multi-Objective Divergence Learning
.png){kind=icon}
PRE
AI
DOI：10.1109/LSP.2021.3077414.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Knowledge distillation has proven to be an effective model compression method that exploits the knowledge from a teacher model for supervising a student model by minimizing the distribution difference between the knowledge and the prediction produced by the student model. In this work, we focus on improving its performance from the perspective of divergence measures. A general form representing a family of divergence measures is introduced and a novel learning paradigm that jointly optimizes multiple measures is proposed by formalizing it as a multi-objective learning problem. Conditioned on Pareto optimality, the weights of different divergences are tuned in an automated way during training. Extensive experiments on multiple datasets show the proposed method can significantly improve the performance of student networks compared to the state-of-the-art methods for knowledge distillation. Codes are available at https://github.com/CMLOO/MoDiv.
Keywords:
Knowledge distillation
multi-objective optimazation
alpha-divergence
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
9.6
Papers:
1.1W
Citations:
1.7W
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSave
ShareSave
View more