.png){kind=icon}
Return
Mini-batch algorithms with online step size
.png){kind=icon}
PRE
AI
DOI：10.1016/j.knosys.2018.11.031.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Mini-batch algorithms have been proposed as a way to speed-up stochastic optimization methods and good results for mini-batch algorithms have been reported previously. A major issue with mini-batch algorithms is how to timely and readily acquire step size while running the algorithm. Usually, mini-batch algorithms employ a diminishing step size, or a best-tuned step size by mentor, which, in practice, are time consuming. To solve this problem, we propose using a hypergradient to compute an online step size (OSS) for mini-batch algorithms. Specifically, we incorporate online step size into advanced mini-batch algorithms, mini-batch nonconvex stochastic variance reduced gradient (MSVRG), thereby generating a new method, MSVRG-OSS. When computing step size in MSVRG-OSS, mini-batch samples are used. In addition, MSVRG-OSS, which needs little additional computation, requires only one extra copy of the original gradient to be stored in memory. We prove that MSVRG-OSS converges linearly in expectation and analyze its complexity. We present numerical results on problems arising with machine learning that indicate the proposed method shows great promise. We also show that, with slightly large batch samples, MSVRG-OSS is insensitive to the initial parameters, which are the key factor for controlling the performance of the algorithm. (C) 2018 Elsevier B.V. All rights reserved.
Keywords:
Stochastic optimization
Convex optimization
Mini-batch of samples
Online step size
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
K
IF:
7.6
Papers:
1.3W
Citations:
4.5W
Organization
Cited Papers
.png){kind=icon}
ShareSave
ShareSaveShareSaveShareSaveShareSave
View more