.png){kind=icon}
.png){kind=icon}
Return
AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation
.png){kind=icon}
OA
AI
DOI：10.1016/j.knosys.2024.111692.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
We present a novel adversarial penalized self-knowledge distillation method, named adversarial learning and implicit regularization for self-knowledge distillation (AI-KD), which regularizes the training procedure by adversarial learning and implicit distillations. Our model not only distills the deterministic and progressive knowledge which are from the pre -trained and previous epoch predictive probabilities but also transfers the knowledge of the deterministic predictive distributions using adversarial learning. The motivation is that the self-knowledge distillation methods regularize the predictive probabilities with soft targets, but the exact distributions may be hard to predict. Our proposed method deploys a discriminator to distinguish the distributions between the pre -trained and student models while the student model is trained to fool the discriminator in the trained procedure. Thus, the student model not only can learn the pre -trained model's predictive probabilities but also align the distributions between the pre -trained and student models. We demonstrate the effectiveness of the proposed method with network architectures on multiple datasets and show the proposed method achieves better performance than existing approaches.
Keywords:
Self-knowledge distillation
Regularization
Adversarial learning
Image classification
Fine-grained dataset
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
K
IF:
7.6
Papers:
1.2W
Citations:
4.5W