Fine-Grained Learning Behavior-Oriented Knowledge Distillation for Graph Neural Networks

Kang Liu

doi:10.1109/TNNLS.2024.3420895

Fine-Grained Learning Behavior-Oriented Knowledge Distillation for Graph Neural Networks

2024-01-01

0

PRE

AI

DOI：10.1109/TNNLS.2024.3420895

原文链接

原文求助

摘要

En 中文

Knowledge distillation (KD), as an effective compression technology, is used to reduce the resource consumption of graph neural networks (GNNs) and facilitate their deployment on resource-constrained devices. Numerous studies exist on GNN distillation, and however, the impacts of knowledge complexity and differences in learning behavior between teachers and students on distillation efficiency remain underexplored. We propose a KD method for fine-grained learning behavior (FLB), comprising two main components: feature knowledge decoupling (FKD) and teacher learning behavior guidance (TLBG). Specifically, FKD decouples the intermediate-layer features of the student network into two types: teacher-related features (TRFs) and downstream features (DFs), enhancing knowledge comprehension and learning efficiency by guiding the student to simultaneously focus on these features. TLBG maps the teacher model's learning behaviors to provide reliable guidance for correcting deviations in student learning. Extensive experiments across eight datasets and 12 baseline frameworks demonstrate that FLB significantly enhances the performance and robustness of student GNNs within the original framework.

Keyword:

Feature knowledge decoupling (FKD)

gradient correction

graph neural networks (GNNs)

knowledge distillation (KD)

learning behavior

Feature knowledge decoupling (FKD)

gradient correction

graph neural networks (GNNs)

knowledge distillation (KD)

learning behavior

Fine-Grained Learning Behavior-Oriented Knowledge Distillation for Graph Neural Networks

摘要

期刊

机构

引用论文