.png){kind=icon}
返回
Accelerating deep neural network training with inconsistent stochastic gradient descent
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neunet.2017.06.003.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Stochastic Gradient Descent (SGD) updates Convolutional Neural Network (CNN) with a noisy gradient computed from a random batch, and each batch evenly updates the network once in an epoch. This model applies the same training effort to each batch, but it overlooks the fact that the gradient variance, induced by Sampling Bias and Intrinsic Image Difference, renders different training dynamics on batches. In this paper, we develop a new training strategy for SGD, referred to as Inconsistent Stochastic Gradient Descent (ISGD) to address this problem. The core concept of ISGD is the inconsistent training, which dynamically adjusts the training effort w.r.t the loss. ISGD models the training as a stochastic process that gradually reduces down the mean of batch's loss, and it utilizes a dynamic upper control limit to identify a large loss batch on the fly. ISGD stays on the identified batch to accelerate the training with additional gradient updates, and it also has a constraint to penalize drastic parameter changes. ISGD is straightforward, computationally efficient and without requiring auxiliary memories. A series of empirical evaluations on real world datasets and networks demonstrate the promising performance of inconsistent training. (C) 2017 Elsevier Ltd. All rights reserved.
Keyword:
Neural networks
Stochastic gradient descent
Statistical process control
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.3
论文数:
8.2K
被引数:
3.0W
机构
引用论文
.png){kind=icon}
Nature-Based Solutions as a Tool in the New Circular Economic Model for Climate Change Adaptation
OAAI
OAAI分享收藏WiFiHonk: Smartphone-Based Beacon Stuffed WiFi Car2X-Communication System for Vulnerable Road User SafetyWiFiHonk: 基于智能手机的信标填充WiFi Car2X-Communication系统，用于脆弱的道路使用者安全
PREAI
分享收藏 没有更多内容
没有更多内容