Residual Learning for Self-Knowledge Distillation: Enhancing Neural Networks Through Consistency Across Layers

doi:10.1109/TBDATA.2025.3552326

Residual Learning for Self-Knowledge Distillation: Enhancing Neural Networks Through Consistency Across Layers

0

PRE

AI

S

Shuoxi Zhang

何

何琨 (Kun He)

DOI：10.1109/TBDATA.2025.3552326

Original

Original request for help

Save

Abstract

En 中文

Knowledge distillation is widely used technique to transfer knowledge from a large pretrained teacher network to a small student network. However, training complex teacher models requires significant computational resources and storage. To address this, a growing area of research, known as self-knowledge distillation (Self-KD), aims to enhance the performance of a neural network by leveraging its own latent knowledge. Despite its potential, existing Self-KD methods often struggle to effectively extract and utilize the model's dark knowledge. In this work, we identify a consistency problem between feature layer and output layer, and propose a novel Self-KD approach called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Residual Learning for <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Self-<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Knowledge <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Distillation (<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RSKD). Our method addresses this issue by enabling the last feature layer of the student model learn the residual gap between the outputs of the pseudo-teacher and the student. Additionally, we extend RSKD by allowing each intermediate feature layer of the student model to learn the residual gap between the corresponding deeper features of the pseudo-teacher and the student. Extensive experiments on various visual datasets demonstrate the effectiveness of the proposed method, which outperforms the state-of-the-art baselines.

Keywords:

Knowledge distillation

self-knowledge distillation

residual learning

feature consistency

Residual Learning for Self-Knowledge Distillation: Enhancing Neural Networks Through Consistency Across Layers

Abstract

Journal

Organization

Cited Papers

Citing Papers