.png){kind=icon}
Return
Difficulty level-based knowledge distillation
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2024.128375.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Knowledge distillation (KD) enables a simple model (student model) to perform as a complex model (teacher model) by distilling the knowledge from a pre-trained teacher model. Existing soft-label distillation methods often use a fixed temperature value in the softmax function to prevent overconfidence in the distillation process. However, this approach can lead to the suppression of important 'dark knowledge' for non-target classes in difficult samples, while also over-smoothing the confidence values for easier samples. To address this issue, we propose a novel approach called difficulty level-based knowledge distillation (DLKD), which considers the difficulty level of each sample to distill refined knowledge with high or low confidence, depending on the sample's complexity. Our method calculates the difficulty level based on the Euclidean distance between the teacher model's predictions and the pruned teacher model's predictions. Experimental results demonstrate that our DLKD method outperforms state-of-the-art methods on challenging samples, including those with noisy labels or augmented data, achieving superior results on CIFAR-100, FGVR, and ImageNet datasets for image classification.
Keywords:
Knowledge distillation
Teacher-student model
Network pruning
Sample difficulty
Image classification
Long-tail sample
Fine-grained visual recognition
Journal
IF:
6.5
Papers:
2.5W
Citations:
6.5W
Organization
No organization information available
Cited Papers
ShareSaveShareSaveShareSaveP-PseudoLabel: Enhanced Pseudo-Labeling Framework With Network Pruning in Semi-Supervised Learning.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAIShareSaveShareSaveShareSave
View more