.png){kind=icon}
.png){kind=icon}
Return
An accelerated stochastic variance-reduced method for machine learning problems
.png){kind=icon}
PRE
AI
DOI：10.1016/j.knosys.2020.105941.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Variance reduction techniques provide simple and fast algorithms for solving machine learning problems. In this paper, we present a novel stochastic variance-reduced method. The proposed method relies on the mini-batch version of stochastic recursive gradient algorithm (MB-SARAH), which updates stochastic gradient estimates by using a simple recursive scheme. However, facing the challenge of the step size sequence selection in MB-SARAH, we introduce an online step size sequence based on the hypergradient descent (HD) method, which only requires little additional computation. For the proposed method, referred to as MB-SARAH-HD, we provide a general convergence analysis and prove linear convergence for strongly convex problems in expectation. Specifically, we prove that the proposed method has sublinear convergence rate in a single outer loop. We also prove that the iteration complexity outperforms several variants of the state-of-the-art stochastic gradient descent (SGD) method under suitable conditions. Numerical experiments on standard datasets are provided to demonstrate the efficacy and superiority of our MB-SARAH-HD method over existing approaches in the literature. (C) 2020 Elsevier B.V. All rights reserved.
Keywords:
Stochastic optimization
Variance reduction
Hypergradient
Recursive gradient
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
K
IF:
7.6
Papers:
1.2W
Citations:
4.5W