Structure-Aware Knowledge Distillation Using Centered Kernel Alignment

Maya Komatsuzaki

doi:10.1117/12.3102573

Structure-Aware Knowledge Distillation Using Centered Kernel Alignment

2026-01-01

0

PRE

AI

DOI：10.1117/12.3102573

Original

Original request for help

Save

Abstract

En 中文

Knowledge distillation (KD) is a powerful technique for model compression, but conventional feature-based methods often overlook global structural consistency. We propose Structure-Aware Knowledge Distillation (StrKD), which explicitly enforces structural correspondence between teacher and student models. By leveraging Centered Kernel Alignment (CKA), Str-KD partitions teacher layers into functional groups based on representational similarity. The student layers are then divided into the same number of groups and aligned with the teacher's structure to ensure holistic consistency. A novel CKA-based loss then encourages the student to mimic the teacher's hierarchical processing. Experiments on CIFAR-100 using VGG and ResNet architectures demonstrate that Str-KD consistently outperforms Vanilla KD and other feature-based baselines. Our results show that incorporating global structural information is crucial for effective and efficient knowledge transfer.

Keywords:

Centered Kernel Alignment

Convolutional Neural Network

Transfer Learning

Knowledge Distillation

Structure-Aware Knowledge Distillation Using Centered Kernel Alignment

Abstract

Journal

Organization