.png){kind=icon}
返回
Analysis of Model Compression Using Knowledge Distillation
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2022.3197608.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In the development of deep learning, several convolution neural network (CNN) models are designed to solve various tasks. However, these CNN models are complex and cumbersome to achieve state-of-the-art performance. The current CNN models remain to suffer from the problem of large models. Thus, model compression techniques are proposed to cope with the complex CNN models. Meanwhile, the selection of compressed model to suit the user requirement significantly contributes during the deployment process. This paper analyses two model compressions, namely the layerwise and the widthwise compression. The compression techniques are implemented in the MobileNetV1 model. Then, knowledge distillation is applied to compensate for the accuracy loss of the compressed model. We demonstrate the analysis of those compressed models from various perspectives and develop several suggestions on the trade-off between the performance and the compression rate. In addition, we also show that the feature that is learned by the compressed models using knowledge distillation has better representation compared to the vanilla model. Our experiment shows that the widthwise compression on MobileNetV1 achieves a compression rate of 42.27% and the layerwise compression achieves 32.42%, respectively. Furthermore, the improvement of the compressed models using knowledge distillation is notable for the widthwise compression with the increasing accuracy above 4.71%.
Keyword:
Convolutional neural networks
Knowledge engineering
Computer architecture
Computational modeling
Training data
Deep learning
Convolutional neural networks
Information filters
Deep learning
knowledge distillation
model compression
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
.png){kind=icon}
分享收藏
A novel and efficient xanthenic dye–organometallic ion‐pair complex for photoinitiating polymerization一种用于光引发聚合的新型高效的黄原胶染料-有机金属离子对配合物
PREAI
分享收藏Recognition of rice leaf diseases and wheat leaf diseases based on multi-task deep transfer learning基于多任务深度迁移学习的水稻叶部病害和小麦叶部病害识别
PREAI
分享收藏分享收藏Structural study of lanthanides(III) in aqueous nitrate and chloride solutions by EXAFS通过EXAFS对硝酸盐和氯化物水溶液中镧系元素 (III) 的结构研究
PREAI
分享收藏Developing Wind and/or Solar Powered Crop Irrigation Systems for the Great Plains为大平原开发风能和/或太阳能作物灌溉系统
PREAI
分享收藏分享收藏
查看更多内容