.png){kind=icon}
Return
BERTBooster: A knowledge enhancement method jointing incremental training and gradient optimization
.png){kind=icon}
OA
AI
DOI：10.1002/int.22998.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The knowledge-enhanced BERT model solves the problem of lacking knowledge in downstream tasks by injecting external expertize, and achieves higher accuracy compared with BERT model. However, owning to large-scale external knowledge is utilized into knowledge-enhanced BERT, some shortcomings comes such as information noise, lower accuracy and weak generalization ability, and so on. To solve this problem, a knowledge enhancement method BERTBooster which combines incremental learning and gradient optimization is proposed. BERTBooster disassembles the input text corpus into entity noun sets through entity noun recognition, and uses the incremental learning task denoising entity auto-encoder to create an incremental task set of entity nouns and external knowledge triples. Furthermore, BERTBooster introduces a new gradient optimization algorithm ChildTuningF into BERT model to improve the generalization ability. BERTBooster can effectively improve the factual knowledge cognition ability of CAGBERT model and improve the accuracy of the model in downstream tasks. Experiments are carried out on six public data sets such as Book_Review, LCQMC, XNLI, Law_QA, Insureace_QA, and NLPCC-DBQA. The experimental results show that the accuracy rate in downstream tasks is increased by 0.65% on average after using BERTBooster on CAGBERT.
Keywords:
gradient optimization
incremental training
knowledge enhanced
natural language processing
pretraining models
Journal
IF:
3.7
Papers:
3.1K
Citations:
8.1K
Organization
Cited Papers
ShareSaveEvaluation of the Approach for the Identification of Trajectory Anomalies on CCTV Video from Road Intersections.png)
MathematicsOAAI
MathematicsIF0
OAAIShareSaveShareSaveKnowledge structure enhanced graph representation learning model for attentive knowledge tracingOAAI
OAAIShareSaveShareSaveShareSave
View more