arrow
返回

Accelerating deep learning with memcomputing

delete2019-02-01
delete22
delete
OA
AI
H
Haik Manukian *
F
Fabio L. Traversa
M
Massimiliano Di Ventra
DOI:10.1016/j.neunet.2018.10.012delete
delete原文链接
delete分享
delete收藏
查看原文
摘要

摘要

En 中文
Restricted Boltzmann machines (RBMs) and their extensions, often called ''deep-belief networks'', are powerful neural networks that have found applications in the fields of machine learning and artificial intelligence. The standard way to train these models resorts to an iterative unsupervised procedure based on Gibbs sampling, called ''contrastive divergence'', and additional supervised tuning via back-propagation. However, this procedure has been shown not to follow any gradient and can lead to suboptimal solutions. In this paper, we show an efficient alternative to contrastive divergence by means of simulations of digital memcomputing machines (DMMs) that compute the gradient of the log-likelihood involved in unsupervised training. We test our approach on pattern recognition using a modified version of the MNIST data set of hand-written numbers. DMMs sample effectively the vast phase space defined by the probability distribution of RBMs, and provide a good approximation close to the optimum. This efficient search significantly reduces the number of generative pretraining iterations necessary to achieve a given level of accuracy in the MNIST data set, as well as a total performance gain over the traditional approaches. In fact, the acceleration of the pretraining achieved by simulating DMMs is comparable to, in number of iterations, the recently reported hardware application of the quantum annealing method on the same network and data set. Notably, however, DMMs perform far better than the reported quantum annealing results in terms of quality of the training. Finally, we also compare our method to recent advances in supervised training, like batch-normalization and rectifiers, that seem to reduce the advantage of pretraining. We find that the memcomputing method still maintains a quality advantage (> 1% in accuracy, corresponding to a 20% reduction in error rate) over these approaches, despite the network pretrained with memcomputing defines a more non-convex landscape using sigmoidal activation functions without batch-normalization. Our approach is agnostic about the connectivity of the network. Therefore, it can be extended to train full Boltzmann machines, and even deep networks at once. (C) 2018 Elsevier Ltd. All rights reserved.
Keyword:
Deep learning
Restricted Boltzmann machines
Memcomputing
AI总结

AI总结

对已上传原文的论文进行重点信息的提取,主要内容包括:简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。

期刊

Neural Networks 封面图
Neural Networks
IF:
6.3
论文数:
8.4K
被引数:
3.9W

机构

U
university of california-san diego
学者数:
20
论文数: 10
被引数: 1
University of California System 封面图
University of California System
学者数:
37.7W
论文数: 33.8W
被引数: 55.6W
引用论文

引用论文

Interaction between the Components of the Interferon γ Receptor Complex
err1995-09-01
err0
errOAAI
errSerguei V. Kotenko; Lara S. Izotova; Brian P. Pollack; Thomas M. Mariano; Robert J. Donnelly; Geetha Muthukumaran; Jeffry R. Cook; Gianni Garotta; Olli Silvennoinen; James N. Ihle; Sidney Pestka
err分享
err收藏
Universal Memcomputing Machines
err2015-11-01
err101
errOAAI
errTraversa, Fabio Lorenzo; Di Ventra, Massimiliano
err分享
err收藏
Solvent effects in liquid-phase dehydration reaction of ethanol to diethylether catalysed by sulfonic-acid catalyst
err2011-02-01
err0
PREAI
errLaurent Vanoye; Marie-Line Zanota; Audrey Desgranges; Alain Favre-Reguillon; Claude De Bellefon
err分享
err收藏
Heterogeneous anchoring in dichotomous choice valuation framework
err2008-01-22
err0
errOAAI
errEmmanuel Flachaire; Guillaume Hollard; Stéphane Luchini
err分享
err收藏
学者 查看更多内容