.png){kind=icon}
返回
Distilling Structural Knowledge from CNNs to Vision Transformers for Data-Efficient Visual Recognition
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neunet.2026.108601.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Knowledge distillation (KD) is an effective strategy to transfer learned representations from a pre-trained teacher model to a smaller student model. Current methods for knowledge transfer from convolutional neural networks (CNNs) to vision transformers (ViTs) mainly align output logits. However, such approaches often overlook the rich semantic structures encoded in CNN features, thereby restricting ViTs from effectively inheriting the inductive biases inherent in convolutional architectures. To this end, this paper proposes a Feature-based CNN-to-ViT Structural Knowledge Distillation framework, dubbed FSKD, which combines the semantic structural knowledge embedded in CNN (teacher) features with the strength of ViT (student) in capturing long-range dependencies. Specifically, this framework includes a feature alignment module to bridge the representational gap between CNN and ViT features, and it incorporates a global feature alignment loss. Additionally, we develop patch-wise and attention-wise distillation losses to transfer inter-patch similarity and attention distribution, facilitating semantic structural knowledge transfer from CNNs to ViTs. Experimental results demonstrate that the proposed method considerably enhances ViT performance in visual recognition tasks, particularly under scenarios with limited data. Code is available at Github .
期刊
IF:
6.3
论文数:
8.2K
被引数:
3.0W
机构
引用论文
分享收藏Cross-modal knowledge distillation for continuous sign language recognition连续手语识别的跨模态知识提炼.png)
NEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
分享收藏分享收藏Irrelevant Patch-Masked Autoencoders for Enhancing Vision Transformers under Limited Data无关补丁掩码自编码器：在有限数据下增强视觉变换器
PREAI
分享收藏Layerwised multimodal knowledge distillation for vision-language pretrained model视觉语言预训练模型的分层多模态知识提炼NEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
分享收藏Multiattentive Perception and Multilayer Transfer Network Using Knowledge Distillation for RGB-D Indoor Scene Parsing多注意力感知与多层级迁移网络，利用知识蒸馏进行RGB-D室内场景解析
PREAI
分享收藏
查看更多内容