.png){kind=icon}
返回
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation
.png){kind=icon}
OA
AI
DOI：10.1162/tacl_a_00517.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Large pretrained language models (PLMs) are often domain- or task-adapted via finetuning or prompting. Finetuning requires modifying all of the parameters and having enough data to avoid overfitting while prompting requires no training and few examples but limits performance. Instead, we prepare PLMs for data- and parameter-efficient adaptation by learning to learn the difference between general and adapted PLMs. This difference is expressed in terms of model weights and sublayer structure through our proposed dynamic low-rank reparameterization and learned architecture controller. Experiments on few-shot dialogue completion, low-resource abstractive summarization, and multi-domain language modeling show improvements in adaptation time and performance over direct finetuning or preparation via domain-adaptive pretraining. Ablations show our task-adaptive reparameterization (TARP) and model search (TAMS) components individually improve on other parameter-efficient transfer like adapters and structure-learning methods like learned sparsification.
期刊
T
IF:
6.9
论文数:
486
被引数:
5.7K
机构
引用论文
Matrix Effects in the Detection of Pb and Ba in Soils Using Laser-Induced Breakdown Spectroscopy使用激光诱导击穿光谱检测土壤中Pb和Ba的基质效应
.png){kind=icon}
PREAI
分享收藏
Geometrically Strained Carbenes: Interdependence among Geometry, Spin Multiplicity, and Reactivity.OAAI
OAAI分享收藏Titanium catalysis for the synthesis of fine chemicals – development and trends钛催化合成精细化学品的发展与趋势
PREAI
分享收藏Phosphate Release Kinetics in Calcareous Grassland and Forest Soils in Response to H+Addition石灰性草地和森林土壤对H 添加的磷酸盐释放动力学
PREAI
分享收藏
查看更多内容