.png){kind=icon}
返回
Heterogeneous Student Knowledge Distillation From BERT Using a Lightweight Ensemble Framework
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2024.3372568.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep learning models have demonstrated their effectiveness in capturing complex relationships between input features and target outputs across many different application domains. These models, however, often come with considerable memory and computational demands, posing challenges for deployment on resource-constrained edge devices. Knowledge distillation is a prominent technique for transferring the expertise from an advanced yet heavy teacher model to a more efficient leaner student model. As ensemble methods have exhibited notable enhancements in model generalization and have achieved state-of-the-art performance in various machine learning tasks, we adopt ensemble techniques to perform knowledge distillation from BERT using multiple lightweight student models. Our approach applies lean architectural paradigms of spatial and sequential networks including LSTM, CNN and their fusion to perform data processing from distinct perspectives. Instead of using contextual word representations which require more space in natural language processing applications, we take advantage of a single static pre-trained and low-dimensional word embedding space to be shared among student models. Empirical studies are conducted on the sentiment classification problem and our model outperforms not only other existing techniques but also the teacher model.
Keyword:
Knowledge distillation
ensemble methods
BERT
LSTM
CNN
contextual word representations
pre-trained and low-dimensional word embedding space
sentiment classification problem
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
Short Text Sentiment Analysis Based on Multi-Channel CNN With Multi-Head Attention Mechanism基于多头注意力机制的多通道CNN短文本情感分析.png)
IEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏
查看更多内容