.png){kind=icon}
返回
Mini-batch algorithms with online step size
.png){kind=icon}
PRE
AI
DOI：10.1016/j.knosys.2018.11.031.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Mini-batch algorithms have been proposed as a way to speed-up stochastic optimization methods and good results for mini-batch algorithms have been reported previously. A major issue with mini-batch algorithms is how to timely and readily acquire step size while running the algorithm. Usually, mini-batch algorithms employ a diminishing step size, or a best-tuned step size by mentor, which, in practice, are time consuming. To solve this problem, we propose using a hypergradient to compute an online step size (OSS) for mini-batch algorithms. Specifically, we incorporate online step size into advanced mini-batch algorithms, mini-batch nonconvex stochastic variance reduced gradient (MSVRG), thereby generating a new method, MSVRG-OSS. When computing step size in MSVRG-OSS, mini-batch samples are used. In addition, MSVRG-OSS, which needs little additional computation, requires only one extra copy of the original gradient to be stored in memory. We prove that MSVRG-OSS converges linearly in expectation and analyze its complexity. We present numerical results on problems arising with machine learning that indicate the proposed method shows great promise. We also show that, with slightly large batch samples, MSVRG-OSS is insensitive to the initial parameters, which are the key factor for controlling the performance of the algorithm. (C) 2018 Elsevier B.V. All rights reserved.
Keyword:
Stochastic optimization
Convex optimization
Mini-batch of samples
Online step size
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
K
IF:
7.6
论文数:
1.3W
被引数:
4.5W
机构
引用论文
Mini-batch algorithms with Barzilai-Borwein update step具有barzilai-borwein更新步骤的小批量算法.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏
查看更多内容