.png){kind=icon}
返回
STKD: Distilling Knowledge From Synchronous Teaching for Efficient Model Compression
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2022.3164264.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Knowledge distillation (KD) transfers discriminative knowledge from a large and complex model (known as teacher) to a smaller and faster one (known as student). Existing advanced KD methods, limited to fixed feature extraction paradigms that capture teacher's structure knowledge to guide the training of the student, often fail to obtain comprehensive knowledge to the student. Toward this end, in this article, we propose a new approach, synchronous teaching knowledge distillation (STKD), to integrate online teaching and offline teaching for transferring rich and comprehensive knowledge to the student. In the online learning stage, a blockwise unit is designed to distill the intermediate-level knowledge and high-level knowledge, which can achieve bidirectional guidance of the teacher and student networks. Intermediate-level information interaction provides more supervisory information to the student network and is useful to enhance the quality of final predictions. In the offline learning stage, the STKD approach applies a pretrained teacher to further improve the performance and accelerate the training process by providing prior knowledge. Trained simultaneously, the student learns multilevel and comprehensive knowledge by incorporating online teaching and offline teaching, which combines the advantages of different KD strategies through our STKD method. Experimental results on the SVHN, CIFAR-10, CIFAR-100, and ImageNet ILSVRC 2012 real-world datasets show that the proposed method achieves significant performance improvements compared with the state-of-the-art methods, especially with satisfying accuracy and model size. Code for STKD is provided at https://github.com/nanxiaotong/STKD.
Keyword:
Knowledge engineering
Training
Computer architecture
Learning systems
Feature extraction
Benchmark testing
Performance evaluation
Knowledge distillation (KD)
model compression
mutual learning network
prior knowledge
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
分享收藏分享收藏Thermal comfort, perceived air quality, and cognitive performance when personally controlled air movement is used by tropically acclimatized persons当热带适应的人使用个人控制的空气运动时，热舒适性，感知的空气质量和认知表现.png)
Indoor Air
OAAI
Indoor AirIF0
OAAI分享收藏Do Perceptions of Competence Mediate The Relationship Between Fundamental Motor Skill Proficiency and Physical Activity Levels of Children in Kindergarten?能力的感知是否可以介导幼儿园儿童的基本运动技能熟练程度与身体活动水平之间的关系？
PREAI
分享收藏
查看更多内容