.png){kind=icon}
返回
An accelerated stochastic variance-reduced method for machine learning problems
.png){kind=icon}
PRE
AI
DOI：10.1016/j.knosys.2020.105941.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Variance reduction techniques provide simple and fast algorithms for solving machine learning problems. In this paper, we present a novel stochastic variance-reduced method. The proposed method relies on the mini-batch version of stochastic recursive gradient algorithm (MB-SARAH), which updates stochastic gradient estimates by using a simple recursive scheme. However, facing the challenge of the step size sequence selection in MB-SARAH, we introduce an online step size sequence based on the hypergradient descent (HD) method, which only requires little additional computation. For the proposed method, referred to as MB-SARAH-HD, we provide a general convergence analysis and prove linear convergence for strongly convex problems in expectation. Specifically, we prove that the proposed method has sublinear convergence rate in a single outer loop. We also prove that the iteration complexity outperforms several variants of the state-of-the-art stochastic gradient descent (SGD) method under suitable conditions. Numerical experiments on standard datasets are provided to demonstrate the efficacy and superiority of our MB-SARAH-HD method over existing approaches in the literature. (C) 2020 Elsevier B.V. All rights reserved.
Keyword:
Stochastic optimization
Variance reduction
Hypergradient
Recursive gradient
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
K
IF:
7.6
论文数:
1.3W
被引数:
4.5W
机构
引用论文
Mini-batch algorithms with Barzilai-Borwein update step具有barzilai-borwein更新步骤的小批量算法.png)
NEUROCOMPUTING
NEUROCOMPUTINGIF6.5
PREAI
分享收藏分享收藏分享收藏
查看更多内容