.png){kind=icon}
返回
BERTBooster: A knowledge enhancement method jointing incremental training and gradient optimization
.png){kind=icon}
OA
AI
DOI：10.1002/int.22998.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The knowledge-enhanced BERT model solves the problem of lacking knowledge in downstream tasks by injecting external expertize, and achieves higher accuracy compared with BERT model. However, owning to large-scale external knowledge is utilized into knowledge-enhanced BERT, some shortcomings comes such as information noise, lower accuracy and weak generalization ability, and so on. To solve this problem, a knowledge enhancement method BERTBooster which combines incremental learning and gradient optimization is proposed. BERTBooster disassembles the input text corpus into entity noun sets through entity noun recognition, and uses the incremental learning task denoising entity auto-encoder to create an incremental task set of entity nouns and external knowledge triples. Furthermore, BERTBooster introduces a new gradient optimization algorithm ChildTuningF into BERT model to improve the generalization ability. BERTBooster can effectively improve the factual knowledge cognition ability of CAGBERT model and improve the accuracy of the model in downstream tasks. Experiments are carried out on six public data sets such as Book_Review, LCQMC, XNLI, Law_QA, Insureace_QA, and NLPCC-DBQA. The experimental results show that the accuracy rate in downstream tasks is increased by 0.65% on average after using BERTBooster on CAGBERT.
Keyword:
gradient optimization
incremental training
knowledge enhanced
natural language processing
pretraining models
期刊
IF:
3.7
论文数:
3.1K
被引数:
8.1K
机构
引用论文
Evaluation of the Approach for the Identification of Trajectory Anomalies on CCTV Video from Road Intersections.png)
MathematicsOAAI
MathematicsIF0
OAAI分享收藏Knowledge structure enhanced graph representation learning model for attentive knowledge tracing面向注意知识追踪的知识结构增强图表示学习模型OAAI
OAAI分享收藏分享收藏
查看更多内容