.png){kind=icon}
返回
Frozen Weights as Prior for Parameter-Efficient Fine-Tuning
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2025.3529807.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In the fields of natural language processing and computer vision, the emergence of large pre-trained models has led to the adoption of fine-tuning them for downstream tasks as an important paradigm. However, the full fine-tuning approach often comes with a hefty cost, which is not feasible for many researchers. Therefore, in recent years, numerous fine-tuning methods have been proposed to efficiently learn incremental updates of pre-trained weights in a more parameter-efficient way (e.g., employing low-rank increments or introducing adapters to modify the network architecture). However, most of these methods involve adding a set of incrementally learned parameters from scratch. From the perspective of full fine-tuning, these approaches often fail to fully exploit the connection between incremental changes during fine-tuning and the frozen weights of the pre-trained model. In order to delve into how to more effectively harness the weights of pre-trained models during the fine-tuning process to fully acquire new knowledge, we propose a novel parameter-efficient approach, which reuses the Frozen Weights as a prior (FoWA). We adapt the incremental matrix to the unitary matrix obtained from the singular value decomposition of the frozen weights, and further fine-tune the model by incorporating prior information. Through the frozen weight prior, FoWA can automatically select the appropriate rank and decouple the number of trainable parameters from the rank. We have conducted extensive experiments on various tasks, including natural language processing, question answering, natural language generation, and visual classification, demonstrating the effectiveness of FoWA.
Keyword:
Adaptation models
Tuning
Matrix decomposition
Vectors
Training
Computational modeling
Singular value decomposition
Transfer learning
Costs
Computer architecture
Fine-tuning
PEFT
large pre-trained models
transfer learning
adaptation
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
.png){kind=icon}
Molecular and Pharmacological Characterization of GABAA Receptors in the Rat Pituitary大鼠垂体中GABAA 受体的分子和药理学特征
PREAI
分享收藏Inflexibility of mental planning: A characteristic disorder with prefrontal lobe lesions?心理计划的僵化: 前额叶病变的特征性障碍？
PREAI
分享收藏分享收藏
Geometrically Strained Carbenes: Interdependence among Geometry, Spin Multiplicity, and Reactivity.OAAI
OAAI分享收藏分享收藏
查看更多内容