.png){kind=icon}
返回
Knowledge Distillation With Feature Self Attention
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2023.3265382.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
With the rapid development of deep learning technology, the size and performance of the network continuously grow, making network compression essential for commercial applications. In this paper, we propose a Feature Self Attention (FSA) module that extracts correlation information between the hidden features of a network and a new method for distilling the correlation features to compress the model. FSA does not require a special module or network to match features between the teacher model and the student model. By removing the multi-head structure and the repeated self-attention blocks in the existing self-attention mechanism, it minimizes the addition of parameters. Based on ResNet-18, 34, the added parameters are only 2.00M and the training speed is also the fastest in comparison to benchmark models. It was demonstrated through experiments that the use of interrelationship loss between features can be beneficial for training student models, indicating the importance of considering correlation information in deep neural network compression. And it was verified through training from scratch on the vanilla without the pre-trained weight of the student model.
Keyword:
Knowledge distillation
self-attention
model compression
training from scratch
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
.png){kind=icon}
分享收藏Research on the Efficiency of Wireless Power Transfer System Based on Multi-Auxiliary Transmitting Coils基于多辅助发射线圈的无线电能传输系统效率研究
PREAI
分享收藏Review of Design Experiments Using the Taguchi Approach: 16 Steps to Product and Process Improvement
PREAI
分享收藏Computations of Standard Binding Free Energies with Molecular Dynamics Simulations用分子动力学模拟计算标准结合自由能
OAAI
OAAI分享收藏
查看更多内容