.png){kind=icon}
返回
Enhancing text classification with attention matrices based on BERT
.png){kind=icon}
PRE
AI
DOI：10.1111/exsy.13512.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Text classification is a critical task in the field of natural language processing. While pre-trained language models like BERT have made significant strides in improving performance in this area, the distinctive dependency information that is present in text has not been fully exploited. Besides, BERT mostly captures phrase-level information in lower layers, which becomes progressively weaker with the increasing depth of layers. To address these limitations, our work focuses on enhancing text classification through the incorporation of Attention Matrices, particularly in the fine-tuning process of pre-trained models like BERT. Our approach, named AM-BERT, leverages learned dependency relationships as external knowledge to enhance the pre-trained model by generating attention matrices. In addition, we introduce a new learning strategy that enables the model to retain learned phrase-level structure information. Extensive experiments and detailed analysis on multiple benchmark datasets demonstrate the effectiveness of our approach in text classification tasks. Furthermore, we show that AM-BERT achieves stable performance improvements also in named entity recognition tasks.
Keyword:
attention matrix
dependency parsing
phrase-level structure information
text classification
期刊
IF:
2.3
论文数:
2.6K
被引数:
3.8K
机构
引用论文
.png){kind=icon}
分享收藏A new low-power, universal, multi-mode Gm-C filter in CNTFET technology一种新型低功耗、通用、多模式Gm-C滤波器，采用CNTFET技术
PREAI
分享收藏Absolute Fluorine Atom Densities in Fluorocarbon High-Density Plasmas Measured by Appearance Mass Spectrometry通过出现质谱法测量的氟碳高密度等离子体中的绝对氟原子密度
PREAI
分享收藏
分享收藏
分享收藏分享收藏
查看更多内容