.png){kind=icon}
返回
Universal embedding for pre-trained models and data bench
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2024.129107.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The transformer architecture has shown significant improvements in the performance of various natural language processing (NLP) tasks. One of the great advantages of transformer-based model is that they allow for the addition of an extra layer to a pre-trained model (PTM) and fine-tuning, rather than requiring the development of a separate architecture for each task. This approach has provided great promising performance in NLP tasks. Therefore, selecting an appropriate PTM from the model zoo, such as Hugging Face, becomes a crucial task. Despite the importance of PTM selection, it still requires further investigation. The main challenge in PTM selection for NLP tasks is the lack of a publicly available benchmark to evaluate model performance for each task and dataset. To address this challenge, we introduce the first public data benchmark to evaluate the performance of popular transformer-based models on diverse ranges of NLP tasks. Furthermore, we propose graph representations of transformer-based models with node features that represent the matrix weight on each layer. Empirical results demonstrate that our proposed graph neural network (GNN) model outperforms existing PTM selection methods.
Keyword:
Transfer learning
Pretrained models
Graph neural networks
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
SBA-15:TiO2 nanocomposites: II. Direct and post-synthesis using acetylacetoneSBA-15:TiO2纳米复合材料: II。使用乙酰丙酮的直接和后合成
PREAI
分享收藏
EPQuant: A Graph Neural Network compression approach based on product quantizationEPQuant: 一种基于乘积量化的图神经网络压缩方法.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏The COMT Val158Met polymorphism regulates the effect of a dopamine antagonist on the feedback‐related negativityCOMT Val158Met多态性调节多巴胺拮抗剂对反馈相关负性的影响
PREAI
分享收藏
查看更多内容