Teacher-student complementary sample contrastive distillation

Zhiqiang Bao

doi:10.1016/j.neunet.2023.11.036

Teacher-student complementary sample contrastive distillation

2024-02-01

2

PRE

AI

DOI：10.1016/j.neunet.2023.11.036

Original

Original request for help

Save

Abstract

En 中文

Knowledge distillation (KD) is a widely adopted model compression technique for improving the performance of compact student models, by utilizing the dark knowledgeof a large teacher model. However, previous studies have not adequately investigated the effectiveness of supervision from the teacher model, and overconfident predictions in the student model may degrade its performance. In this work, we propose a novel framework, Teacher-Student Complementary Sample Contrastive Distillation (TSCSCD), that alleviate these challenges. TSCSCD consists of three key components: Contrastive Sample Hardness (CSH), Supervision Signal Correction (SSC), and Student Self-Learning (SSL). Specifically, CSH evaluates the teacher's supervision for each sample by comparing the predictions of two compact models, one distilled from the teacher and the other trained from scratch. SSC corrects weak supervision according to CSH, while SSL employs integrated learning among multi-classifiers to regularize overconfident predictions. Extensive experiments on four real-world datasets demonstrate that TSCSCD outperforms recent state-of-the-art knowledge distillation techniques.

Keywords:

Knowledge distillation

Transfer learning

Model regularization

Sample hardness

Deep learning

Teacher-student complementary sample contrastive distillation

Abstract

Journal

Organization