.png){kind=icon}
返回
Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution
.png){kind=icon}
OA
AI
DOI：10.1109/TNNLS.2022.3186807.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Knowledge distillation (KD) is an effective framework that aims to transfer meaningful information from a large teacher to a smaller student. Generally, KD often involves how to define and transfer knowledge. Previous KD methods often focus on mining various forms of knowledge, for example, feature maps and refined information. However, the knowledge is derived from the primary supervised task, and thus, is highly task-specific. Motivated by the recent success of self-supervised representation learning, we propose an auxiliary self-supervision augmented task to guide networks to learn more meaningful features. Therefore, we can derive soft self-supervision augmented distributions as richer dark knowledge from this task for KD. Unlike previous knowledge, this distribution encodes joint knowledge from supervised and self-supervised feature learning. Beyond knowledge exploration, we propose to append several auxiliary branches at various hidden layers, to fully take advantage of hierarchical feature maps. Each auxiliary branch is guided to learn self-supervision augmented tasks and distill this distribution from teacher to student. Overall, we call our KD method a hierarchical self-supervision augmented KD (HSSAKD). Experiments on standard image classification show that both offline and online HSSAKD achieves state-of-the-art performance in the field of KD. Further transfer experiments on object detection further verify that HSSAKD can guide the network to learn better features. The code is available at https://github.com/winycg/HSAKD.
Keyword:
Task analysis
Knowledge engineering
Self-supervised learning
Feature extraction
Probability distribution
Training
Semantics
Knowledge distillation
representation learning
self-supervised learning
visual recognition
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
分享收藏Multi-View correlation distillation for incremental object detection用于增量目标检测的多视图相关性提取.png)
PATTERN RECOGNITION
OAAI
PATTERN RECOGNITIONIF7.6
OAAI分享收藏分享收藏
查看更多内容