.png){kind=icon}
返回
Memory efficient data-free distillation for continual learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2023.109875.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep neural networks suffer from the catastrophic forgetting phenomenon when trained on sequential tasks in continual learning, especially when data from previous tasks are unavailable. To mitigate catastrophic forgetting, various methods either store data from previous tasks, which may raise privacy concerns, or require large memory storage. Particularly, the distillation-based methods mitigate catastrophic forgetting by using proxy datasets. However, proxy datasets may not match the distributions of the original datasets of previous tasks. To address these problems in a setting where the full training data of previous tasks are unavailable and memory resources are limited, we propose a novel data-free distillation method. Our method encodes knowledge of previous tasks into network parameter gradients by Taylor expansion, deducing a regularizer relying on gradients in network training loss. To improve memory efficiency, we design an approach to compressing the gradients in the regularizer. Moreover, we theoretically analyze the approximation error of our method. Experimental results on multiple datasets demonstrate that our proposed method outperforms the existing approaches in continual learning.
Keyword:
Continual learning
Catastrophic forgetting
Knowledge distillation
期刊
IF:
7.6
论文数:
1.3W
被引数:
4.5W
机构
引用论文
Exemplar-free class incremental learning via discriminative and comparable parallel one-class classifiers.png)
PATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
分享收藏FoCL: Feature-oriented continual learning for generative modelsFoCL: 生成模型的面向特征的持续学习PATTERN RECOGNITION
OAAI
PATTERN RECOGNITIONIF7.6
OAAI分享收藏分享收藏分享收藏 没有更多内容
没有更多内容