.png){kind=icon}
.png){kind=icon}
Return
Joint structured pruning and dense knowledge distillation for efficient transformer model compression
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2021.05.084.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In this paper, we develop a novel Joint Model Compression (referred to as JMC) method by combining structured pruning and dense knowledge distillation techniques to significantly compress original large language model into a deep compressed shallow network. In particular, a new Direct Importance-aware Structured Pruning (referred as DISP) approach is proposed to structurally prune the redundant structures in the Transformer networks directly based on the corresponding parameter matrices in the model. Besides, a Dense Knowledge Distillation (referred to as DKD) method is developed with a many-to-one layer mapping strategy to leverage more comprehensive layer-wise linguistic knowledge for the distillation. Further, the proposed structured pruning and dense knowledge distillation are integrated together to perform the joint compression, which enables us to achieve a significant compression without sacrificing model accuracy. The extensive experimental results across four NLP tasks on seven datasets demonstrate its effectiveness and superiority to the baselines, while maintaining similar performance to original large model with further remarkable benefits for inference-time speedup and memory efficiency. (c) CO 2021 Elsevier B.V. All rights reserved.
Keywords:
Transformer Model Compression
Structured Pruning
Knowledge Distillation
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
6.5
Papers:
2.5W
Citations:
6.5W