.png){kind=icon}
.png){kind=icon}
Return
Data-Dependent Convergence for Consensus Stochastic Optimization
.png){kind=icon}
PRE
AI
DOI：10.1109/TAC.2017.2671377.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
We study a distributed consensus-based stochastic gradient descent (SGD) algorithm and show that the rate of convergence involves the spectral properties of two matrices: The standard spectral gap of a weight matrix from the network topology and a new term depending on the spectral norm of the sample covariance matrix of the data. This data-dependent convergence rate shows that distributed SGD algorithms perform better on datasets with small spectral norm. Our analysis method also allows us to find data-dependent convergence rates as we limit the amount of communication. Spreading a fixed amount of data across more nodes slows convergence; for asymptotically growing datasets, we show that addingmoremachines can help when minimizing twice-differentiable losses.
Keywords:
Convergence
distributed computing
machine learning
minimization
optimization
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
7
Papers:
1.3W
Citations:
6.7W