.png){kind=icon}
.png){kind=icon}
Return
Adaptive Temperature Distillation method for mining hard samples' knowledge
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2025.129745.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Knowledge distillation can transfer knowledge from a complex teacher network into a simple student one through a high temperature factor, improving the latter's performance. However, existing studies usually use fixed temperatures, making them ineffective in mining the rich knowledge contained in hard samples. Specifically, high temperature tends to over-smooth knowledge on hard samples, whereas low temperature makes knowledge almost equivalent to hard labels on easy samples. In this paper, we propose an Adaptive Temperature Distillation (ATD) method to effectively address these challenges. A well-trained teacher network's information entropy is used to assess a sample's relative difficulty. Then, low temperature is used in a hard sample, which allows the student network to learn its dark knowledge more effectively. And high temperature is employed in an easy sample to prevent the student network from becoming overconfident and ignoring the dark knowledge of negative classes. Furthermore, we propose a mixup variant to enable the student network to access more hard samples with rich dark knowledge. Instead of focusing on data augmentation as the existing mixup studies, ATD pays attention to increasing the richness of dark knowledge by mixing the output logits of easy and hard samples. The overall performance of ATD is verified in multiple benchmark datasets by comparing it with state-of-the-art knowledge distillation methods.
Keywords:
Knowledge distillation
Knowledge mining
Mixup
Temperature
Journal
IF:
6.5
Papers:
2.5W
Citations:
6.5W