.png){kind=icon}
Return
Memory efficient data-free distillation for continual learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.patcog.2023.109875.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deep neural networks suffer from the catastrophic forgetting phenomenon when trained on sequential tasks in continual learning, especially when data from previous tasks are unavailable. To mitigate catastrophic forgetting, various methods either store data from previous tasks, which may raise privacy concerns, or require large memory storage. Particularly, the distillation-based methods mitigate catastrophic forgetting by using proxy datasets. However, proxy datasets may not match the distributions of the original datasets of previous tasks. To address these problems in a setting where the full training data of previous tasks are unavailable and memory resources are limited, we propose a novel data-free distillation method. Our method encodes knowledge of previous tasks into network parameter gradients by Taylor expansion, deducing a regularizer relying on gradients in network training loss. To improve memory efficiency, we design an approach to compressing the gradients in the regularizer. Moreover, we theoretically analyze the approximation error of our method. Experimental results on multiple datasets demonstrate that our proposed method outperforms the existing approaches in continual learning.
Keywords:
Continual learning
Catastrophic forgetting
Knowledge distillation
Journal
IF:
7.6
Papers:
1.3W
Citations:
4.5W
Organization
Cited Papers
Exemplar-free class incremental learning via discriminative and comparable parallel one-class classifiers.png)
PATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
ShareSave
ShareSaveShareSaveShareSaveShareSaveShareSave no more
no more