Encoding-based memory for recurrent neural networks

Antonio Carta

doi:10.1016/j.neucom.2021.04.051

Return

Encoding-based memory for recurrent neural networks

2021-10-01

PRE

Antonio Carta *

Alessandro Sperduti

Davide Bacciu

DOI：10.1016/j.neucom.2021.04.051

Original

Original request for help

Save

Abstract

En 中文

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study memorization from the point of view of the design and training of recurrent neural networks. We study how to maximize the short-term memory of recurrent units, an objective difficult to achieve using backpropagation. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences. Additionally, we provide a specialized training algorithm that initializes the memory to efficiently encode the hidden activations of the network. Experimental results on synthetic and real-world datasets show that the chosen encoding mechanism is superior to static encodings such as orthogonal models and the delay line. The method also outperforms RNN and LSTM units trained using stochastic gradient descent. Experiments on symbolic music modeling show that the training algorithm specialized for the memorization component improves the final performance compared to stochastic gradient descent. (c) 2021 Elsevier B.V. All rights reserved.

Keywords:

Recurrent neural networks

Autoencoders

Linear dynamical systems

Memory capacity

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.