.png){kind=icon}
Return
Collaborative knowledge distillation via filter knowledge transfer
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2023.121884.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Knowledge distillation is a promising model compression technique that generally distills the knowledge from a complex teacher model to a lightweight student model. However, the performance gain of a student model is usually limited by the capacity gap between the large teacher model and the small student model. In this paper, we propose a new collaborative knowledge distillation method that makes use of a new strategy, named Filter Knowledge Transfer (FKT) to detect and learn the valuable filter information following from the teacher to the student. To be specific, the useful knowledge of filters measured by using information entropy is transferred between different peer networks and unimportant filters are reactivated according to the ratio based on the characterized filter information entropy during the online distillation process. Experimental results derived on four popular datasets, CIFAR-10/100, Market-1501, and Tiny-ImageNet, demonstrate the superiority of our proposed method over the others we considered.
Keywords:
Model compression
Knowledge distillation
Filter pruning
Filter knowledge transfer
Journal
IF:
7.5
Papers:
3.0W
Citations:
10.2W
Organization
View more organizations
Cited Papers
.png){kind=icon}
ShareSaveShareSave
ShareSave
no more