.png){kind=icon}
返回
Hyperbolic Pre-Trained Language Model
.png){kind=icon}
PRE
AI
DOI：10.1109/TASLP.2024.3407575.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In recent years, we have witnessed significant improvements in pre-trained language models (PLM) brought about by the scaling of parameter sizes and data amounts. However, this also brings high computational and storage costs. In this paper, we present a new direction to improve PLMs without scaling parameters and data: adopting a geometric feature space that is more suitable for encoding the intrinsic structured features of text. Although text is generally considered unstructured data, it possesses rich intrinsic structured features that signify syntactic and semantic relationships. Leveraging these structured features is vital for text understanding. Given that structured features are better encoded in hyperbolic spaces than in the Euclidean spaces used by conventional PLMs, we propose that PLMs should operate entirely within hyperbolic spaces. Our experiments demonstrate the superiority of hyperbolic PLMs over Euclidean PLMs across a wide variety of tasks, using the same parameter and data settings. This suggests that altering the geometry of model representation is a promising direction for model enhancement.
Keyword:
Pre-trained language model
fine-tuning
hyperbolic geometry
期刊
I
IF:
5.1
论文数:
2.6K
被引数:
1.1W
机构
引用论文
.png){kind=icon}
The effects of nanoemulsions on the fatty acid profiles of sea bass fillets during storage at 2±2 °C
OAAI
OAAI分享收藏
分享收藏
查看更多内容