.png){kind=icon}
返回
On the step size selection in variance-reduced algorithm for nonconvex optimization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2020.114336.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Nonconvex problems have recently received considerable attention in statistics, machine learning and vision areas. Variance reduction techniques like the SVRG-type and SARAH-type methods, originally proposed for convex optimization, have been proved to be effective for nonconvex optimization. The performance of variance-reduced stochastic gradient methods greatly depends on how step sizes are tuned and decreased over time. However, research on specifying online step size in nonconvex optimization is quite limited. Motivated by this gap, we propose using the random stabilized Barzilai-Borwein (RSBB) method to compute step size for variance-reduced stochastic gradient methods. In particular, we incorporate RSBB into the modern variance-reduced stochastic gradient method, the SARAH-type method, for nonconvex optimization. We theoretically prove that the proposed method converges sub-linearly for general nonconvex objective functions and converges linearly for gradient dominated functions. We also show that the complexity of our proposed method is comparable to modern stochastic gradient methods. To further show the efficacy of the RSBB method, we proposed MSVRG-RSBB, by introducing it into MSVRG (a variant of the SVRG method). We provide experimental results on various datasets to show the efficacy of our proposed methods.
Keyword:
Nonconvex optimization
Variance reduction
Online step size
Stochastic optimization
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
7.5
论文数:
3.0W
被引数:
10.2W
机构
引用论文
A survey of internet marketing by small and medium-sized enterprises for placing wine on the market
PREAI
分享收藏
分享收藏Mini-batch algorithms with Barzilai-Borwein update step具有barzilai-borwein更新步骤的小批量算法.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏Solution of nonconvex and nonsmooth economic dispatch by a new Adaptive Real Coded Genetic Algorithm
PREAI
分享收藏分享收藏
查看更多内容