返回
Accelerating deep learning with memcomputing
DOI:10.1016/j.neunet.2018.10.012.png)
摘要
En 中文
Restricted Boltzmann machines (RBMs) and their extensions, often called ''deep-belief networks'', are powerful neural networks that have found applications in the fields of machine learning and artificial intelligence. The standard way to train these models resorts to an iterative unsupervised procedure based on Gibbs sampling, called ''contrastive divergence'', and additional supervised tuning via back-propagation. However, this procedure has been shown not to follow any gradient and can lead to suboptimal solutions. In this paper, we show an efficient alternative to contrastive divergence by means of simulations of digital memcomputing machines (DMMs) that compute the gradient of the log-likelihood involved in unsupervised training. We test our approach on pattern recognition using a modified version of the MNIST data set of hand-written numbers. DMMs sample effectively the vast phase space defined by the probability distribution of RBMs, and provide a good approximation close to the optimum. This efficient search significantly reduces the number of generative pretraining iterations necessary to achieve a given level of accuracy in the MNIST data set, as well as a total performance gain over the traditional approaches. In fact, the acceleration of the pretraining achieved by simulating DMMs is comparable to, in number of iterations, the recently reported hardware application of the quantum annealing method on the same network and data set. Notably, however, DMMs perform far better than the reported quantum annealing results in terms of quality of the training. Finally, we also compare our method to recent advances in supervised training, like batch-normalization and rectifiers, that seem to reduce the advantage of pretraining. We find that the memcomputing method still maintains a quality advantage (> 1% in accuracy, corresponding to a 20% reduction in error rate) over these approaches, despite the network pretrained with memcomputing defines a more non-convex landscape using sigmoidal activation functions without batch-normalization. Our approach is agnostic about the connectivity of the network. Therefore, it can be extended to train full Boltzmann machines, and even deep networks at once. (C) 2018 Elsevier Ltd. All rights reserved.
Keyword:
Deep learning
Restricted Boltzmann machines
Memcomputing
AI总结
对已上传原文的论文进行重点信息的提取,主要内容包括:简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.3
论文数:
8.4K
被引数:
3.9W
机构
引用论文
Improved Particle Size Control for the Dispersion Polymerization of Methyl methacrylate in Supercritical Carbon Dioxide超临界二氧化碳中甲基丙烯酸甲酯分散聚合的改进粒度控制
Study of bi-directional buck-boost converter topologies for application in electrical vehicle motor drives应用于电动汽车电机驱动的双向buck-boost变换器拓扑研究
Two-dimensional bricklayer arrangements of tolans using halogen bonding interactions使用卤素键相互作用的tolans的二维瓦工层布置

