LaRA: Layer-wise rank allocation for efficient fine-tuning of pruned large language models

Yuhua Zhou

doi:10.1016/j.ipm.2025.104538

LaRA: Layer-wise rank allocation for efficient fine-tuning of pruned large language models

2025-12-11

0

PRE

AI

Y

Yuhua Zhou

C

Changhai Zhou

S

Shiyang Zhang

F

Fei Yang

Y

Yi Zhang

A

Aimin Pan

DOI：10.1016/j.ipm.2025.104538

Original

Original request for help

Save

Abstract

En 中文

• Introduce LaRA, an automated framework that restores pruned LLMs via LoRA by assigning layer-specific ranks, achieving optimal performance with minimal overhead. • Propose a Rank Allocation Model, trained with policy-gradient reinforcement learning, and a Score Model environment that evaluates rank configurations as rewards. • Build a diverse dataset of architecture-accuracy pairs spanning pruning levels, LLM families, and ranks to robustly train both models. • Extensive experiments across tasks, pruning settings, and LLMs validate LaRA’s effectiveness and generality

LaRA: Layer-wise rank allocation for efficient fine-tuning of pruned large language models

Abstract

Journal

Organization

Cited Papers

Citing Papers