.png){kind=icon}
.png){kind=icon}
Return
MDIKD: multi-dimensional integration knowledge distillation
.png){kind=icon}
PRE
AI
DOI：10.1007/s00530-025-02061-4.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Knowledge distillation is an effective model compression and acceleration technique in the field of deep learning, and the mainstream methods include logit-based, feature-based, and relational. Feature and relation distillation has better performance due to the introduction of more teacher information, but the extraction of deep features during training requires additional computation and storage, and the cost is higher. Traditional logit distillation is easy to implement, but expresses abstract and high-level information, and knowledge transfer is not sufficient. We found that the final logit of the teacher model contains multiple categories of information, and the weight distribution of the fully-connected layer leads to these information having different responses in different dimensions. The multi-dimensional features can be decomposed into different dimensional logit features through dimensionality reduction, which captures category differences and relationships in more detail, but the range and distribution of the values of these features varies greatly due to the difference in semantic relevance between categories, which affects the distillation efficiency. For this reason, dimensional feature normalisation is used to enable different dimensional logit features to be compared for learning on a more consistent distribution, further improving performance. Experiments show that on the CIFAR-100 dataset, the accuracy of heterogeneous teacher-student pairs is improved by 6.69% and isomorphic teacher-student pairs is improved by 3.18% compared with the baseline, and the performance gap with mainstream feature distillation is only 0.21%, while the training efficiency is more than twice as much as it is, which verifies that this method can significantly enhance distillation under the premise of guaranteeing the efficiency by refining the logit knowledge representation.
Keywords:
Deep learning
Knowledge distillation
Model compression
Image classification