.png){kind=icon}
返回
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
.png){kind=icon}
OA
AI
DOI：10.1162/tacl_a_00413.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resourcehungry and computation-intensive to suit lowcapability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.
期刊
T
IF:
6.9
论文数:
486
被引数:
5.7K
机构
引用论文
Metal to Insulator Transition in Epitaxial Graphene Induced by Molecular Doping分子掺杂诱导外延石墨烯的金属-绝缘体转变OAAI
OAAI分享收藏分享收藏分享收藏Mitochondrial Dynamics Associated with Oxygen-Glucose Deprivation in Rat Primary Neuronal Cultures.png)
PLoS ONE
OAAI
PLoS ONEIF0
OAAI分享收藏
查看更多内容