.png){kind=icon}
返回
Parameter-efficient online knowledge distillation for pretrained language models
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2024.126040.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
With the advancement of natural language processing (NLP), the size of datasets and pre-trained language models (PLMs) has exponentially grown. These vast models exhibit robust capabilities in generation, comprehension, and multimodal processing. However, the complexity of parameters poses challenges for real-time and edge computing applications. Knowledge distillation (KD) has emerged as a common approach to address this issue by transferring prior knowledge from a large teacher to a smaller student model. Previous KD methods relied on offline full-parameter fine-tuning distillation, where the primary objective of the teacher model was to optimize downstream tasks, often neglecting the focus on KD itself. Such inefficient KD methods limit the potential of both teacher and student models. This study introduces a parameter-efficient online distillation method, PEKD, to tackle these challenges. In contrast to previous KD approaches, PEKD streamlines the distillation process into a single training stage and leverages parallel adapters to enhance fine-tuning efficiency. The teacher and student models, utilizing only 0.5% of the parameters on downstream tasks, achieve or surpass the accuracy of full fine-tuning methods. Extensive experiments on the GLUE benchmark dataset demonstrate the competitiveness of PEKD. The code for this study is accessible at https://github.com/Kawlyh/PEKD.
Keyword:
Online knowledge distillation
Pre-trained language model
Parameter compression
期刊
IF:
7.5
论文数:
3.0W
被引数:
10.2W
机构
暂无机构信息
引用论文
The Impact of Artificial Intelligence on Language Translation: A Review人工智能对语言翻译的影响综述.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAI分享收藏A Deep Learning-Based Approach for Part of Speech (PoS) Tagging in the Pashto LanguageIEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches, and Future Directions抽象对话总结的分类: 场景，方法和未来方向OAAI
OAAI分享收藏Base on contextual phrases with cross-correlation attention for aspect-level sentiment analysis基于具有互相关注意的上下文短语的方面级情感分析
PREAI
分享收藏 没有更多内容
没有更多内容