.png){kind=icon}
返回
Meta-learning for compressed language model: A multiple choice question answering study
.png){kind=icon}
OA
AI
DOI：10.1016/j.neucom.2021.01.148.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Model compression is a promising approach for reducing the model size of pretrained-language-models (PLMs) on low resource edge devices and applications. Unfortunately, the compression process always accompanies a cost of performance degradation, especially for the low resource downstream tasks, i.e., multiple-choice question answering. To address the degradation issue of model compression on PLMs, we proposed an end-to-end reptile (ETER) meta-learning approach to improving the performance of PLMs on the low resource multiple-choice question answering task. Specifically, our ETER improves the traditional two-stage meta-learning to an end-to-end manner, integrating the target finetuning stage into the meta training stage. To strengthen the generic meta-learning, ETER employs two-level meta-task construction from instance-level and domain-level to enrich its task generalization. What is more, ETER optimizes meta-learning by parameter constraints to reduce its parameter learning space. Experiments demonstrate that ETER significantly improved the performance of compressed PLMs and achieved large superiority over the baselines on different datasets.(c) 2021 Elsevier B.V. All rights reserved.
Keyword:
End-to-end reptile
Compressed pretrained-language-model
Meta-learning
Multiple-choice question answering
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
Sodium channel kinetic changes that produce Brugada syndrome or progressive cardiac conduction system disease产生Brugada综合征或进行性心脏传导系统疾病的钠通道动力学变化
PREAI
分享收藏分享收藏
Mitochondrial Dynamics Associated with Oxygen-Glucose Deprivation in Rat Primary Neuronal Cultures.png)
PLoS ONEOAAI
PLoS ONEIF0
OAAI分享收藏
查看更多内容