.png){kind=icon}
返回
Residual error based knowledge distillation
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2020.10.113.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Knowledge distillation (KD) is one of the most popular ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To remedy this problem, this paper proposes Residual error based Knowledge Distillation (RKD), which further distills the knowledge by introducing an assistant model(A). Specifically, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. In this way, S and A complement with each other to get better knowledge from T. Furthermore, we devise an effective method to derive S and A from a given model without increasing the total computational cost. Extensive experiments show that our approach achieves appealing results on popular classification data sets, CIFAR-100 and ImageNet, surpassing state-of-the-art methods and keep strong robustness to adversarial samples. CO 2020 Published by Elsevier B.V.
Keyword:
Model compression
Knowledge distillation
Residual learning
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
Welding characteristics of aluminum, copper, nickel and aluminum alloy with alumina coating using ultrasonic complex vibration welding equipments铝、铜、镍及铝合金氧化铝涂层超声复合振动焊接特性研究
PREAI
分享收藏A survey of deep neural network architectures and their applications深度神经网络体系结构及其应用综述.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏 没有更多内容
没有更多内容