Exploiting deep representations for natural language processing

Zi-Yi Dou

doi:10.1016/j.neucom.2019.12.060

Exploiting deep representations for natural language processing

2020-04-01

8

PRE

AI

DOI：10.1016/j.neucom.2019.12.060

原文链接

原文求助

摘要

En 中文

Advanced neural network models generally implement systems as multiple layers to model complex functions and capture complicated linguistic structures at different levels [1]. However, only the top layers of deep networks are leveraged in the subsequent process, which misses the opportunity to exploit the useful information embedded in other layers. In this work, we propose to expose all of these embedded signals with two types of mechanisms, namely deep connections and iterative routings. While deep connections allow better information and gradient flow across layers, iterative routings directly combine the layer representations to form a final output with iterative routing-by-agreement mechanism. Experimental results on both machine translation and language representation tasks demonstrate the effectiveness and universality of the proposed approaches, which indicates the necessity of exploiting deep representations for natural language processing tasks. While the two strategies individually boost performance, combining them can further improve performance. (c) 2019ElsevierB.V. Allrightsreserved.

Keyword:

Natural language processing

Deep neural networks

Deep representations

Layer aggregation

Routing-by-agreement

Exploiting deep representations for natural language processing

摘要

AI总结

期刊

机构

引用论文