.png){kind=icon}
Return
A layered self-supervised knowledge distillation framework for efficient image representation learning
.png){kind=icon}
PRE
AI
DOI：10.1007/s00607-026-01671-8.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deploying deep learning models in resource-constrained environments requires training strategies that improve compact models without increasing inference-time cost. This work addresses this challenge through an efficient self-distillation framework. We propose Layered Self-Supervised Knowledge Distillation (LSSKD), a training framework that enhances compact deep neural networks without relying on large pre-trained teacher models. The method introduces auxiliary classifiers at intermediate network stages to generate self-supervised knowledge and enable hierarchical knowledge transfer across layers. Experiments on CIFAR-100, Tiny-ImageNet, and ImageNet demonstrate that LSSKD consistently improves the performance of compact models. The method achieves an average improvement of 4.54% over PS-KD and a 1.14% gain over SSKD on CIFAR-100, while improving performance on ImageNet by 0.32% compared with HASSKD. Additional evaluations under few-shot settings further demonstrate the effectiveness of the Proposed LSSKD framework. These findings demonstrate that the Proposed LSSKD framework effectively improves the generalization and performance of compact models without relying on large over-parameterized teacher networks. Importantly, auxiliary classifiers are used only during training and can be removed at inference time, introducing no additional computational cost. This design makes the approach well suited for improving compact vision models intended for resource-constrained deployment scenarios.
Keywords:
Self-supervised learning
Knowledge distillation
Edge computing
Multi-modal learning
Lightweight deep models
Journal
C
IF:
2.8
Papers:
2.3K
Citations:
3.5K
Organization
Cited Papers
ShareSaveScale fusion light CNN for hyperspectral face recognition with knowledge distillation and attention mechanism.png)
APPLIED INTELLIGENCE
APPLIED INTELLIGENCEIF3.5
PREAI
ShareSaveShareSaveShareSave
ShareSave
ShareSave
View more