.png){kind=icon}
返回
Highlight Every Step: Knowledge Distillation via Collaborative Teaching
.png){kind=icon}
OA
AI
DOI：10.1109/TCYB.2020.3007506.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
High storage and computational costs obstruct deep neural networks to be deployed on resource-constrained devices. Knowledge distillation (KD) aims to train a compact student network by transferring knowledge from a larger pretrained teacher model. However, most existing methods on KD ignore the valuable information among the training process associated with training results. In this article, we provide a new collaborative teaching KD (CTKD) strategy which employs two special teachers. Specifically, one teacher trained from scratch (i.e., scratch teacher) assists the student step by step using its temporary outputs. It forces the student to approach the optimal path toward the final logits with high accuracy. The other pretrained teacher (i.e., expert teacher) guides the student to focus on a critical region that is more useful for the task. The combination of the knowledge from two special teachers can significantly improve the performance of the student network in KD. The results of experiments on CIFAR-10, CIFAR-100, SVHN, Tiny ImageNet, and ImageNet datasets verify that the proposed KD method is efficient and achieves state-of-the-art performance.
Keyword:
Training
Knowledge engineering
Neural networks
Collaboration
Computational modeling
Task analysis
Computer vision
deep learning
knowledge distillation (KD)
neural-networks compression
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
10.5
论文数:
1.1W
被引数:
5.0W
机构
引用论文
Measurements of fatty acid synthesis by incorporation of deuterium from deuterated water.png)
Biochemistry
BiochemistryIF0
PREAI
分享收藏A novel and efficient xanthenic dye–organometallic ion‐pair complex for photoinitiating polymerization一种用于光引发聚合的新型高效的黄原胶染料-有机金属离子对配合物
PREAI
分享收藏Intrapartum‐related perinatal deaths in births planned in midwifery‐led settings in Great Britain: findings and recommendations from the ESMiE confidential enquiry英国计划在助产士主导的环境中分娩的与肿瘤相关的围产期死亡: ESMiE机密调查的发现和建议OAAI
OAAI分享收藏
查看更多内容