.png){kind=icon}
.png){kind=icon}
Return
Knowledge fusion distillation and gradient-based data distillation for class-incremental learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2024.129286.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deep neural networks, despite their exceptional performance on individual tasks, face the challenge of catastrophic forgetting when incrementally learning new scenarios. Existing class-incremental learning methods have shown promising results by rehearsing past samples from auxiliary memory. However, preserving only a limited subset of previous samples fails to capture the complete distribution, leading to a decline in memory quality as new data is added. Additionally, issues like exemplar distribution collapse and task-recency bias impede the effective transfer of knowledge from teacher to student during distillation. To address these problems, we propose a novel replay-based learning framework called Knowledge fusion Distillation and gradient-based Data Distillation (K3D). K3D simultaneously optimizes both the classifier and exemplars during the learning phase. By parameterizing exemplars, we enable their optimization through gradient-based data distillation, enhancing their representativeness instead of discarding them, as done in other replay strategies. Furthermore, we enhance the classifier using knowledge fusion distillation, ensuring that decision boundaries are maintained across all classes. Extensive experiments on the CIFAR100 and ImageNet100 benchmarks show that synthetic exemplars optimized by K3D are more representative than selected ones. By combining optimizable exemplars and knowledge fusion distillation, K3D outperforms several state-of-the-art methods and effectively mitigates catastrophic forgetting in class-incremental learning scenarios.
Keywords:
Gradient-based data distillation
Optimizable exemplars
Synthetic replay
Knowledge fusion distillation
Class-incremental learning
Journal
IF:
6.5
Papers:
2.5W
Citations:
6.5W
Organization
View more organizations