.png){kind=icon}
返回
Domain Word Extension Using Curriculum Learning
.png){kind=icon}
OA
AI
DOI：10.3390/s23063064.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Self-supervised learning models, such as BERT, have improved the performance of various tasks in natural language processing. Although the effect is reduced in the out-of-domain field and not the the trained domain thus representing a limitation, it is difficult to train a new language model for a specific domain since it is both time-consuming and requires large amounts of data. We propose a method to quickly and effectively apply the pre-trained language models trained in the general domain to a specific domain's vocabulary without re-training. An extended vocabulary list is obtained by extracting a meaningful wordpiece from the training data of the downstream task. We introduce curriculum learning, training the models with two successive updates, to adapt the embedding value of the new vocabulary. It is convenient to apply because all training of the models for downstream tasks are performed in one run. To confirm the effectiveness of the proposed method, we conducted experiments on AIDA-SC, AIDA-FC, and KLUE-TC, which are Korean classification tasks, and subsequently achieved stable performance improvement.
Keyword:
token expansion
curriculum learning
pre-trained models
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
3.5
论文数:
7.2W
被引数:
20.9W
机构
引用论文
.png){kind=icon}
BatteryBERT: A Pretrained Language Model for Battery Database EnhancementBatteryBERT: 一种用于电池数据库增强的预训练语言模型OAAI
OAAI分享收藏分享收藏Evaluation of the Environmental Performance of Traditional Leather Materials in Fashion Industry
OAAI
OAAI分享收藏分享收藏 没有更多内容
没有更多内容