Explainability-based knowledge distillation

doi:10.1016/j.patcog.2024.111095

Explainability-based knowledge distillation

2025-03-01

0

PRE

AI

DOI：10.1016/j.patcog.2024.111095

Original

Original request for help

Save

Abstract

En 中文

Knowledge distillation (KD) is a popular approach for deep model acceleration. Based on the knowledge distilled, we categorize KD methods as label-related and structure-related. The former distills the very abstract (high-level) knowledge, e.g., logits; and the latter uses the spatial (low- or medium-level feature) knowledge. However, existing KD methods are usually not explainable, i.e., we do not know what knowledge is transferred during distillation. In this work, we propose a new KD method, Explainability-based Knowledge Distillation (Exp-KD). Specifically, we propose to use class activation map (CAM) as the explainable knowledge which can effectively capture both label- and structure-related information during the distillation. We conduct extensive experiments, including image classification tasks on CIFAR-10, CIFAR-100 and ImageNet datasets, and explainability tests on ImageNet and ImageNet-Segmentation. The results show the great effectiveness and explainability of Exp-KD compared with the state-of-the-art. Code is available at https://github.com/ Blenderama/Exp-KD.

Keywords:

Explainability

Knowledge distillation

CAM

Explainability-based knowledge distillation

Abstract

Journal

Organization