Collaborative knowledge distillation via filter knowledge transfer

Jianping Gou

doi:10.1016/j.eswa.2023.121884

Collaborative knowledge distillation via filter knowledge transfer

2024-03-01

5

PRE

AI

DOI：10.1016/j.eswa.2023.121884

Original

Original request for help

Save

Abstract

En 中文

Knowledge distillation is a promising model compression technique that generally distills the knowledge from a complex teacher model to a lightweight student model. However, the performance gain of a student model is usually limited by the capacity gap between the large teacher model and the small student model. In this paper, we propose a new collaborative knowledge distillation method that makes use of a new strategy, named Filter Knowledge Transfer (FKT) to detect and learn the valuable filter information following from the teacher to the student. To be specific, the useful knowledge of filters measured by using information entropy is transferred between different peer networks and unimportant filters are reactivated according to the ratio based on the characterized filter information entropy during the online distillation process. Experimental results derived on four popular datasets, CIFAR-10/100, Market-1501, and Tiny-ImageNet, demonstrate the superiority of our proposed method over the others we considered.

Keywords:

Model compression

Knowledge distillation

Filter pruning

Filter knowledge transfer

Collaborative knowledge distillation via filter knowledge transfer

Abstract

Journal

Organization