.png){kind=icon}
返回
Knowledge fusion distillation and gradient-based data distillation for class-incremental learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2024.129286.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep neural networks, despite their exceptional performance on individual tasks, face the challenge of catastrophic forgetting when incrementally learning new scenarios. Existing class-incremental learning methods have shown promising results by rehearsing past samples from auxiliary memory. However, preserving only a limited subset of previous samples fails to capture the complete distribution, leading to a decline in memory quality as new data is added. Additionally, issues like exemplar distribution collapse and task-recency bias impede the effective transfer of knowledge from teacher to student during distillation. To address these problems, we propose a novel replay-based learning framework called Knowledge fusion Distillation and gradient-based Data Distillation (K3D). K3D simultaneously optimizes both the classifier and exemplars during the learning phase. By parameterizing exemplars, we enable their optimization through gradient-based data distillation, enhancing their representativeness instead of discarding them, as done in other replay strategies. Furthermore, we enhance the classifier using knowledge fusion distillation, ensuring that decision boundaries are maintained across all classes. Extensive experiments on the CIFAR100 and ImageNet100 benchmarks show that synthetic exemplars optimized by K3D are more representative than selected ones. By combining optimizable exemplars and knowledge fusion distillation, K3D outperforms several state-of-the-art methods and effectively mitigates catastrophic forgetting in class-incremental learning scenarios.
Keyword:
Gradient-based data distillation
Optimizable exemplars
Synthetic replay
Knowledge fusion distillation
Class-incremental learning
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
.png){kind=icon}
A double-gradient structured hydrogel for an efficient moisture-electric generator一种双梯度结构水凝胶用于高效湿电发电机
PREAI
分享收藏
查看更多内容