.png){kind=icon}
返回
GRID: Gradient Routing With In-Network Aggregation for Distributed Training
.png){kind=icon}
PRE
AI
DOI：10.1109/TNET.2023.3244794.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
As the scale of distributed training increases, it brings huge communication overhead in clusters. Some works try to reduce the communication cost through gradient com-pression or communication scheduling. However, these methods either downgrade the training accuracy or do not reduce the total transmission amount. One promising approach, called in-network aggregation, is proposed to mitigate the bandwidth bottleneck in clusters by aggregating gradients in programmable hardware (e.g., Intel Tofino switches). However, existing solu-tions mainly implement in-network aggregation through fixed (or default) routing paths, resulting in load imbalancing and long communication time. To deal with this issue, we propose GRID, the first-of-its-kind work on Gradient Routing with In-network Aggregation for Distributed Training. In the control plane, we present an efficient gradient routing algorithm based on randomized rounding and formally analyze the approxi-mation performance. In the data plane, we realize in-network aggregation by carefully designing the logic of workers and programmable switches. We implement GRID and evaluate its performance on a small-scale testbed consisting of 3 Intel Tofino switches and 9 commodity servers. With a combination of testbed experiments and large-scale simulations, we show that GRID can reduce the communication time by 38.4%-60.1% and speed up distributed training by 17.4%-52.7% compared with state-of -the-art solutions.
Keyword:
In-network aggregation
gradient routing
distributed training
datacenter network
programmable network
期刊
I
IF:
3.6
论文数:
4.4K
被引数:
9.5K
机构
引用论文
Vinyl polymers containing amido and carboxylic groups as side substituents: I. Synthesis of N-acryloyl-glycine and N-acryloyl-6-caproic acid and their grafting on cellulose membranes.png)
Polymer
PolymerIF0
PREAI
分享收藏
分享收藏
The Solubility of Hydrous Thorium(IV) Oxide in Chloride Media: Development of an Aqueous Ion-Interaction Modelract
ractIF0
PREAI
分享收藏
查看更多内容