Reconstruct the Pruned Model Without Retraining

Pingjie Wang

doi:10.1109/JSTSP.2025.3568224

2025-05-12

PRE

DOI：10.1109/JSTSP.2025.3568224

原文链接

原文求助

摘要

En 中文

结构化剪枝是一种适用于大型语言模型（LLMs）的硬件友好的压缩技术，其有望实现免重新训练以避免巨大的重新训练成本。这种免重新训练范式涉及剪枝标准来定义架构和失真重建来恢复性能。然而，现有的重建算法通常表现出有限的泛化能力，并可能导致显著的误差累积。为解决这些问题，我们提出了一个基于线性插值的自适应重建（LIAR）框架。通过将线性插值应用于保留的权重，LIAR最小化累积误差并实现高效且准确的重建。此外，LIAR与多种剪枝标准和模块兼容。我们在GLUE、SQuAD、WikiText和推理基准上的评估表明，LIAR使BERT模型在移除50%参数后仍能保持98%的准确率，并在50%的剪枝比例下，在1分钟内为LLaMA-7B带来2.56×的性能提升。

Keyword:

Structured pruning

retraining-free compression

distortion reconstruction

期刊

IEEE Journal of Selected Topics in Signal Processing

IF:

13.7

论文数:

1.9K

被引数:

1.1W

机构

Shanghai Jiao Tong University

学者数:

7.8K

论文数: 2.4K

被引数: 14.8W

引用论文

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation SemEval-2017任务1: 语义文本相似性多语言和跨语种聚焦评价

Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)

IF0

2017-01-01

OAAI

Daniel Cer; Mona Diab; Eneko Agirre; Inigo Lopez-Gazpio; Lucia Specia

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank 情感树库上语义组合性的递归深度模型

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing

IF0

2013-01-01

PREAI

Richard Socher; Alex Perelygin; Jean Wu; Jason Chuang; Christopher D. Manning; Andrew Ng; Christopher Potts

The PASCAL Recognising Textual Entailment Challenge

Lecture Notes in Computer Science

IF0

2006-01-01

PREAI

Ido Dagan; Oren Glickman; Bernardo Magnini

EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference

MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture

IF0

2021-10-17

OAAI

Thierry Tambe; Coleman Hooper; Lillian Pentecost; Tianyu Jia; En-Yu Yang; Marco Donato; Victor Sanh; Paul Whatmough; Alexander M. Rush; David Brooks; Gu-Yeon Wei

Building a Large Annotated Corpus of English: The Penn Treebank

IF0

1993-04-30

OAAI

Mitch Marcus

Attention Is All You Need 注意是你所需要的

IF0

OAAI

Fluctuation-Based Adaptive Structured Pruning for Large Language Models

Proceedings of the AAAI Conference on Artificial Intelligence

IF0

2024-03-24