.png){kind=icon}
返回
Distributed Encoding and Updating for SAZD Coded Distributed Training
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2023.3276888.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Linear combination (LC) based coded distributed computing (CDC) suffers from the problem of poor numerical stability. Therefore, LC-CDC based model parallel (MP) training for a deep nueral network (DNN) may have poor accuracy. To enhance accuracy, we propose to replace LC by shift-and-addition (SA) and replace matrix inversion by zigzag decoding (ZD) in the encoding and decoding process of each layer, respectively, and call the scheme Naive SAZD-CDC based MP training (N-SAZD-CDC-MP-T). However, N-SAZD-CDC-MP-T encounters the problem of bottleneck at the master node, which is caused by frequent encoding/decoding at the master node and frequent huge volume of data delivery between master and worker node. This bottleneck problem may pull down the training speed significantly. To alleviate this bottleneck problem, we further design an enhanced version, by offloading certain processing from master node to distributed encoding and updating (DEU) at the worker nodes and call it DEU-SAZD-CDC-MP-T. A proof that DEU-SAZD-CDC-MP-T automatically maitains the code structure during each iteration is provided. Extensive numerical studies show that the prediction accuracy of SAZD-CDC-MP-T improves significantly over that of Poly (which is representative of LC) based scheme. In addition, the training speed of DEU-SAZD-CDC-MP-T over N-SAZD-CDC-MP-T is improved significantly.
Keyword:
Training
Encoding
Decoding
Data models
Numerical stability
Computational modeling
Numerical models
Coded distributed computing (CDC)
distributed encoding and updating (DEU)
distributed training
shift-and-addition
zigzag decoding
期刊
IF:
6
论文数:
5.2K
被引数:
1.1W
机构
引用论文
.png){kind=icon}
分享收藏Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation具有分层异步模型更新和时间加权聚合的高效通信联合深度学习
OAAI
OAAI分享收藏Collaborate Edge and Cloud Computing With Distributed Deep Learning for Smart City Internet of Things通过分布式深度学习协作边缘和云计算，实现智慧城市物联网
PREAI
分享收藏Two-dimensional bricklayer arrangements of tolans using halogen bonding interactions使用卤素键相互作用的tolans的二维瓦工层布置
PREAI
分享收藏
查看更多内容