.png){kind=icon}
.png){kind=icon}
Return
Regularizing transformers with deep probabilistic layers
.png){kind=icon}
OA
AI
DOI：10.1016/j.neunet.2023.01.032.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Language models (LM) have grown non-stop in the last decade, from sequence-to-sequence archi-tectures to attention-based Transformers. However, regularization is not deeply studied in those structures. In this work, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer. We study its advantages regarding the depth where it is placed and prove its effectiveness in several scenarios. Experimental result demonstrates that the inclusion of deep generative models within Transformer-based architectures such as BERT, RoBERTa, or XLM-R can bring more versatile models, able to generalize better and achieve improved imputation score in tasks such as SST-2 and TREC or even impute missing/noisy words with richer text.(c) 2023 Elsevier Ltd. All rights reserved.
Keywords:
Natural language processing
Regularization
Deep learning
Transformers
Variational auto -encoder
Missing data
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
6.3
Papers:
7.8K
Citations:
3.0W