.png){kind=icon}
返回
DSAG: A Mixed Synchronous-Asynchronous Iterative Method for Straggler-Resilient Learning
.png){kind=icon}
OA
AI
DOI：10.1109/TCOMM.2022.3227286.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
We consider straggler-resilient learning. In many previous works, e.g., in the coded computing literature, straggling is modeled as random delays that are independent and identically distributed between workers. However, in many practical scenarios, a given worker may straggle over an extended period of time. We propose a latency model that captures this behavior and is substantiated by traces collected on Microsoft Azure, Amazon Web Services (AWS), and a small local cluster. Building on this model, we propose DSAG, a mixed synchronous-asynchronous iterative optimization method, based on the stochastic average gradient (SAG) method, that combines timely and stale results. We also propose a dynamic load-balancing strategy to further reduce the impact of straggling workers. We evaluate DSAG for principal component analysis, cast as a finite-sum optimization problem, of a large genomics dataset, and for logistic regression on a cluster composed of 100 workers on AWS, and find that DSAG is up to about 50% faster than SAG, and more than twice as fast as coded computing methods, for the particular scenario that we consider.
Keyword:
Radio frequency
Computational modeling
Principal component analysis
Iterative methods
Stochastic processes
Logistics
Convergence
Coded computing
iterative optimization
load-balancing
principal component analysis (PCA)
stochastic average gradient (SAG)
straggler mitigation
variance reduction
期刊
IF:
8.3
论文数:
1.2W
被引数:
3.6W
机构
引用论文
Searching for exotic particles in high-energy physics with deep learning用深度学习寻找高能物理中的奇异粒子.png)
NATURE COMMUNICATIONS
NATURE COMMUNICATIONSIF15.7
PREAI
分享收藏
分享收藏A Cyclic Carbanionic Valence Isomer of a Carbocation: Diphosphino Analogs of DiaminocarbocationsScience
ScienceIF0
PREAI
分享收藏
查看更多内容