.png){kind=icon}
返回
Difficulty level-based knowledge distillation
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2024.128375.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Knowledge distillation (KD) enables a simple model (student model) to perform as a complex model (teacher model) by distilling the knowledge from a pre-trained teacher model. Existing soft-label distillation methods often use a fixed temperature value in the softmax function to prevent overconfidence in the distillation process. However, this approach can lead to the suppression of important 'dark knowledge' for non-target classes in difficult samples, while also over-smoothing the confidence values for easier samples. To address this issue, we propose a novel approach called difficulty level-based knowledge distillation (DLKD), which considers the difficulty level of each sample to distill refined knowledge with high or low confidence, depending on the sample's complexity. Our method calculates the difficulty level based on the Euclidean distance between the teacher model's predictions and the pruned teacher model's predictions. Experimental results demonstrate that our DLKD method outperforms state-of-the-art methods on challenging samples, including those with noisy labels or augmented data, achieving superior results on CIFAR-100, FGVR, and ImageNet datasets for image classification.
Keyword:
Knowledge distillation
Teacher-student model
Network pruning
Sample difficulty
Image classification
Long-tail sample
Fine-grained visual recognition
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
暂无机构信息
引用论文
P-PseudoLabel: Enhanced Pseudo-Labeling Framework With Network Pruning in Semi-Supervised LearningP-伪标签: 半监督学习中带有网络剪枝的增强型伪标签框架.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAI分享收藏
查看更多内容