.png){kind=icon}
.png){kind=icon}
Return
Knowledge distillation from relative distribution
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2025.127736.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The knowledge distillation method based on logits aims to extract more effective information from the output logits of neural networks, transforming the output generated by the fully connected layer of the teacher model into a probability distribution through softmax operation, and aligning the probability allocation of the student model with that of the teacher model. However, the insensitivity of softmax to negative values results in the absence of negative value information, hindering the transfer of knowledge. To tackle this challenge, we propose a method of logit distillation using relative distributions. Specifically, we first construct Negative Value Reversal Logits (NVRL) from the logits of network outputs to extract information from negative logits, and then use the relative distributions between different classes to supplement the lost distribution information in NVRL. Compared to traditional knowledge distillation methods, our approach achieves improvements ranging from 1.1% to 3.98% on CIFAR-100 and ImageNet datasets, along with a 1.95% improvement in transfer learning tasks. Compared to feature-driven knowledge distillation methods, it does not require additional auxiliary modules for feature matching and can be easily suitable for diverse neural network structures.
Keywords:
Deep learning
Knowledge distillation
Logit knowledge distillation
Relative distribution
Transfer learning
Journal
IF:
7.5
Papers:
2.9W
Citations:
10.2W