.png){kind=icon}
返回
Tolerant Self-Distillation for image classification
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neunet.2024.106215.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep neural networks tend to suffer from the overfitting issue when the training data are not enough. In this paper, we introduce two metrics from the intra-class distribution of correct -predicted and incorrect -predicted samples to provide a new perspective on the overfitting issue. Based on it, we propose a knowledge distillation approach without pretraining a teacher model in advance named Tolerant Self -Distillation (TSD) for alleviating the overfitting issue. It introduces an online updating memory and selectively stores the class predictions of the samples from the past iterations, making it possible to distill knowledge across the iterations. Specifically, the class predictions stored in the memory bank serve as the soft labels for supervising the samples from the same class for the current iteration in a reverse way, i.e. the correct -predicted samples are supervised with the incorrect predictions while the incorrect -predicted samples are supervised with the correct predictions. Consequently, the premature convergence issue caused by the over -confident samples would be mitigated, which helps the model to converge to a better local optimum. Extensive experimental results on several image classification benchmarks, including small-scale, large-scale, and fine-grained datasets, demonstrate the superiority of the proposed TSD.
Keyword:
Overfitting
Self-Distillation
Tolerant
Deep Learning
期刊
IF:
6.3
论文数:
8.2K
被引数:
3.0W
机构
引用论文
.png){kind=icon}
Conductivity Enhancement in Thin Silicon-on-Insulator Layer Embedding Artificial Dislocation Network
PREAI
分享收藏
查看更多内容