.png){kind=icon}
Return
Knowledge distillation via instance-level sequence learning
.png){kind=icon}
OA
AI
DOI：10.1016/j.knosys.2021.107519.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Recently, distillation approaches for extracting general knowledge from a teacher network to guide a student network have been suggested. Most existing methods transfer knowledge from the teacher to the student network by feeding a sequence of random minibatches sampled uniformly from the data. We argue that, instead, a compact student network should be guided gradually using samples ordered in a meaningful sequence. Thus, the gap in feature representation between the teacher and student network can be bridged step by step. In this paper, we provide a curriculum learning knowledge distillation framework via instance-level sequence learning. It employs the student network of the early epoch as a snapshot to create a curriculum for the student network's next training phase. We performed extensive experiments using the CIFAR-10, CIFAR-100, SVHN, and CINIC-10 datasets. When compared with several state-of-the-art methods, our framework achieved the best performance with fewer iterations. (C) 2021 Elsevier B.V. All rights reserved.
Keywords:
Neural networks compression
Knowledge distillation
Computer vision
Deep learning
Journal
K
IF:
7.6
Papers:
1.3W
Citations:
4.5W
Organization
Cited Papers
.png){kind=icon}
ShareSaveGEV-NN: A deep neural network architecture for class imbalance problem in binary classification
PREAI
ShareSaveShareSaveShareSaveShareSave
View more