.png){kind=icon}
Return
Word embedding factor based multi-head attention
.png){kind=icon}
OA
AI
DOI：10.1007/s10462-025-11115-y.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The natural language processing (NLP) field has made significant progress using deep learning models based on multi-head attention mechanisms, such as Transformer and BERT. However, there are two major limitations to this approach. First, the number of heads is often manually set based on empirical experience, and second, it is not clear enough in semantic understanding and interpretation. In this study, we propose a novel attention mechanism called Factor Analysis-based Multi-head (FAM) Attention, which combines the theory of explorative factor analysis and word embedding. The experimental results demonstrate that FAM Attention achieves better performance and requires fewer parameters compared to traditional methods while also having better semantic understanding ability and interpretability at the token level. This also has significant implications for current Large Language Models (LLMs), particularly in terms of effectively reducing parameter counts and enhancing performance.
Keywords:
Multi-head attention
Word embedding
Factor analysis
BERT
Journal
IF:
13.9
Papers:
6.1K
Citations:
1.9W
Organization
Cited Papers
ShareSave
ShareSaveShareSaveShareSaveShareSaveShareSaveShareSaveSeformer: a long sequence time-series forecasting model based on binary position encoding and information transfer regularization.png)
APPLIED INTELLIGENCE
APPLIED INTELLIGENCEIF3.5
PREAI
ShareSave
View more