Prototypical recurrent unit

doi:10.1016/j.neucom.2018.05.048

Return

Prototypical recurrent unit

2018-10-01

Dingkun Long

Richong Zhang *

Yongyi Mao

DOI：10.1016/j.neucom.2018.05.048

Original

Save

View PDF

Abstract

En 中文

Despite the great successes of deep learning, the effectiveness of deep neural networks, such as LSTM/GRU-like recurrent networks, has not been well understood. Not only attributed to their nonlinear dynamics, the difficulty in understanding LSTM/GRU-like recurrent networks also resides in the highly complex recurrence structure in these networks. This work aims at constructing an alternative recurrent unit that is as simple as possible and yet also captures the key components of LSTM/GRU recurrent units. Such a unit, if available, can then be used as a prototype for the study of LSTM/GRU-like networks and potentially enable easier analysis. Towards that goal, we take a system-theoretic perspective to design a new recurrent unit, which we call the prototypical recurrent unit (PRU). Not only having minimal complexity, PRU is demonstrated experimentally to have comparable performance to GRU and LSTM over a range of modelling tasks. This establishes PRU networks as a prototypical example for future study of LSTM/GRU-like recurrent networks. The complexity advantage of PRU may also make it a favourable alternative to LSTM and GRU in practice. (C) 2018 Elsevier B.V. All rights reserved.

Keywords:

Recurrent unit

Recurrent network

State space

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.