Residual error based knowledge distillation

Mengya Gao

doi:10.1016/j.neucom.2020.10.113

Return

Residual error based knowledge distillation

2021-04-01

PRE

DOI：10.1016/j.neucom.2020.10.113

Original

Original request for help

Save

Abstract

En 中文

Knowledge distillation (KD) is one of the most popular ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To remedy this problem, this paper proposes Residual error based Knowledge Distillation (RKD), which further distills the knowledge by introducing an assistant model(A). Specifically, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. In this way, S and A complement with each other to get better knowledge from T. Furthermore, we devise an effective method to derive S and A from a given model without increasing the total computational cost. Extensive experiments show that our approach achieves appealing results on popular classification data sets, CIFAR-100 and ImageNet, surpassing state-of-the-art methods and keep strong robustness to adversarial samples. CO 2020 Published by Elsevier B.V.

Keywords:

Model compression

Knowledge distillation

Residual learning

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.