.png){kind=icon}
返回
A Unified Continual Learning Framework with General Parameter-Efficient Tuning
.png){kind=icon}
OA
AI
DOI：10.1109/ICCV51070.2023.01055.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The pre-training. downstream adaptation presents both new opportunities and challenges for Continual Learning (CL). Although the recent state-of-the-art in CL is achieved through Parameter-Efficient-Tuning (PET) adaptation paradigm, only prompt has been explored, limiting its application to Transformers only. In this paper, we position prompting as one instantiation of PET, and propose a unified CL framework with general PET, dubbed as Learning-Accumulation-Ensemble (LAE). PET, e.g., using Adapter, LoRA, or Prefix, can adapt a pre-trained model to downstream tasks with fewer parameters and resources. Given a PET method, our LAE framework incorporates it for CL with three novel designs. 1) Learning: the pre-trained model adapts to the new task by tuning an online PET module, along with our adaptation speed calibration to align different PET modules, 2) Accumulation: the task-specific knowledge learned by the online PET module is accumulated into an offline PET module through momentum update, 3) Ensemble: During inference, we respectively construct two experts with online/offline PET modules (which are favored by the novel/historical tasks) for prediction ensemble. We show that LAE is compatible with a battery of PET methods and gains strong CL capability. For example, LAE with Adaptor PET surpasses the prior state-of-the-art by 1.3% and 3.6% in last-incremental accuracy on CIFAR100 and ImageNet-R datasets, respectively. Code is available at https://github.com/gqk/LAE.
期刊
I
IF:
0
论文数:
1
被引数:
0
机构
引用论文
Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏WHY THERE ARE COMPLEMENTARY LEARNING-SYSTEMS IN THE HIPPOCAMPUS AND NEOCORTEX - INSIGHTS FROM THE SUCCESSES AND FAILURES OF CONNECTIONIST MODELS OF LEARNING AND MEMORY为什么在海马和新大脑皮层中存在互补的学习系统-来自学习和记忆的连接主义模型的成功和失败的见解.png)
PSYCHOLOGICAL REVIEW
PSYCHOLOGICAL REVIEWIF5.8
PREAI
分享收藏Energy-Efficient Robot Configuration and Motion Planning Using Genetic Algorithm and Particle Swarm Optimization基于遗传算法和粒子群算法的节能机器人配置和运动规划EnergiesOAAI
EnergiesIF0
OAAI分享收藏The adsorption of monolayer coatings on iron nanoparticles: Mössbauer spectroscopy and XANES results
PREAI
分享收藏Porous TiO2 thin film-based photocatalytic windows for an enhanced operation of optofluidic microreactors in CO2 conversioniScience
OAAI
iScienceIF0
OAAI分享收藏
查看更多内容