Expandable Residual Approximation for Knowledge Distillation

doi:10.1109/TNNLS.2025.3602118

Expandable Residual Approximation for Knowledge Distillation

2025-09-08

0

PRE

AI

DOI：10.1109/TNNLS.2025.3602118

Original

Original request for help

Save

Abstract

En

Knowledge distillation (KD) aims to transfer knowledge from a large-scale teacher model to a lightweight one, significantly reducing computational and storage requirements. However, the inherent learning capacity gap between the teacher and student often hinders the sufficient transfer of knowledge, motivating numerous studies to address this challenge. Inspired by the progressive approximation principle in the Stone–Weierstrass theorem, we propose expandable residual approximation (ERA), a novel KD method that decomposes the approximation of residual knowledge into multiple steps, reducing the difficulty of mimicking the teacher’s representation through a divide-and-conquer approach. Specifically, ERA employs a multibranched residual network (MBRNet) to implement this residual knowledge decomposition. Additionally, a teacher weight integration (TWI) strategy is introduced to mitigate the capacity disparity by reusing the teacher’s head weights. Extensive experiments show that ERA improves the Top-1 accuracy on ImageNet classification benchmark by 1.41% and the AP on the MS COCO object detection benchmark by 1.40, as well as achieving leading performance across computer vision tasks.

Keywords:

Image classification

knowledge distillation (KD)

residual learning

semantic segmentation

visual object detection

Expandable Residual Approximation for Knowledge Distillation

Abstract

Journal

Organization