.png){kind=icon}
返回
Adaptive Semiparametric Language Models
.png){kind=icon}
OA
AI
DOI：10.1162/tacl_a_00371.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our model uses extended short-term context by caching local hidden states-similar to transformer-XL-and global long-term memory by retrieving a set of nearest neighbor tokens at each timestep. We design a gating function to adaptively combine multiple information sources to make a prediction. This mechanism allows the model to use either local context, short-term memory, or long-term memory (or any combination of them) on an ad hoc basis depending on the context. Experiments on word-based and character-based language modeling datasets demonstrate the efficacy of our proposed method compared to strong baselines.
期刊
T
IF:
6.9
论文数:
486
被引数:
5.7K
机构
暂无机构信息
.png){kind=icon}
没有更多内容