.png){kind=icon}
返回
Adaptive lightweight network construction method for Self-Knowledge Distillation
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2025.129477.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Self-Knowledge Distillation (self-KD) has become a promising method for neural network compression due to its advances in computational efficiency. Nevertheless, its applicability is constrained by the inherent inflexibility of the network architecture and the absence of quantitative metrics to evaluate the distillability of the architecture. To address these problems, a two-stage adaptive dynamic distillation network framework (ADDN) is proposed to adapt the architecture based on the distillability of the current architecture, containing a hypernetwork topology construction stage and a subnetwork training stage. To evaluate the distillability of current architectures without necessitating extensive training, we propose a set of low-cost distillability metrics that evaluate architectures from the perspective of architectural similarity and clustering ability. Furthermore, to simplify the hypernetwork structure and reduce the complexity of the construction process, a hierarchical filtering module is introduced to dynamically refine and remove candidate operations within the architecture incrementally, contingent upon the distillability of the current architecture. To validate the effectiveness of our approach, we conduct extensive experiments on various image classification datasets and compare with current works. Experimental results demonstrate that the self-knowledge distillation network architecture obtained by our proposed methodology simultaneously attains superior distillability and efficiency while significantly curtailing construction expenses.
Keyword:
Deep learning
Knowledge Distillation
Neural network architecture design
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
.png){kind=icon}
Calcium-modulating cyclophilin ligand regulates membrane trafficking of postsynaptic GABAA receptors
OAAI
OAAI分享收藏分享收藏分享收藏
查看更多内容