.png){kind=icon}
.png){kind=icon}
Return
M3LoRA: Flexible Task Adaptation via Multiple Low-Rank Matrices With Mixture-of-Subspaces and Minor Singular Components Initialization
.png){kind=icon}
OA
AI
X
Y
C
Y
Y
DOI：10.1049/cit2.70144.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Parameter-efficient fine-tuning (PEFT) has become a crucial paradigm for domain adaptation, achieving strong performance by updating only a small fraction of model parameters. Among various PEFT methods, low-rank adaptation (LoRA) is widely adopted due to its structural simplicity and computational efficiency. However, in multitask scenarios, LoRA often suffers from performance degradation due to task interference. Recent extensions, such as mixture-of-experts (MoE) and asymmetric LoRA variants, attempt to mitigate these issues; however, their reliance on fixed subspace-mixing strategies limits flexibility and makes the models more sensitive to input noise and data sparsity. Moreover, vanilla LoRA typically initialises low-rank matrices with Gaussian noise or zeros and optimises them in unconstrained subspaces, which may disrupt the structured representations learnt by pretrained models. In this article, we propose M3LoRA, a novel PEFT framework that leverages multiple low-rank matrices with mixture-of-subspaces and minor singular components initialisation. M3LoRA utilises multiple low-rank matrices to minimise interference between task-specific subspaces while maintaining representational capacity. Furthermore, it employs a learnable mixing matrix positioned between the down-projection and up-projection matrices to dynamically combine their subspaces, thereby enabling adaptive task-specific combination mechanisms. Additionally, it initialises these low-rank matrices within a subspace orthogonal to the principal singular components of pretrained weights—termed the minor singular components—thereby leveraging directions unexplored during pretraining to better capture task-specific features from labelled data. Extensive experiments on a wide range of benchmark datasets demonstrate that M3LoRA achieves substantial improvements over existing PEFT baselines, particularly in multi-task scenarios where task interference often degrades conventional LoRA performance.
Keywords:
low-rank adaptation
mixture-of-expert
parameter-efficient fine-tuning
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
7.3
Papers:
649
Citations:
2.4K
Organization
Cited Papers