.png){kind=icon}
Return
Robust Student Network Learning
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2019.2929114.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deep neural networks bring in impressive accuracy in various applications, but the success often relies on heavy network architectures. Taking well-trained heavy networks as teachers, classical teacher-student learning paradigm aims to learn a student network that is lightweight yet accurate. In this way, a portable student network with significantly fewer parameters can achieve considerable accuracy, which is comparable to that of a teacher network. However, beyond accuracy, the robustness of the learned student network against perturbation is also essential for practical uses. Existing teacher-student learning frameworks mainly focus on accuracy and compression ratios, but ignore the robustness. In this paper, we make the student network produce more confident predictions with the help of the teacher network, and analyze the lower bound of the perturbation that will destroy the confidence of the student network. Two important objectives regarding prediction scores and gradients of examples are developed to maximize this lower bound, to enhance the robustness of the student network without sacrificing the performance. Experiments on benchmark data sets demonstrate the efficiency of the proposed approach to learning robust student networks that have satisfying accuracy and compact sizes.
Keywords:
Perturbation methods
Convolution
Neural networks
Training
Redundancy
Feature extraction
Learning systems
Deep learning
knowledge distillation (KD)
teacher-student learning
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
8.9
Papers:
7.5K
Citations:
7.2W
Organization
Cited Papers
A Hierarchical Cardiac Rhythm Classification Methodology Based on Electrocardiogram Fiducial PointsOAAI
OAAIShareSaveSynthesis of n-type semiconducting diamond film using diphosphorus pentaoxide as the doping source
PREAI
ShareSaveShareSaveShareSaveShareSaveShareSaveInactivation of the 20S proteasome in Streptomyces lividans and its influence on the production of heterologous proteins.png)
MicrobiologyOAAI
MicrobiologyIF0
OAAIShareSaveReview of Design Experiments Using the Taguchi Approach: 16 Steps to Product and Process Improvement
PREAI
ShareSaveShareSave
View more