.png){kind=icon}
返回
Accelerating Distributed Training With Collaborative In-Network Aggregation
.png){kind=icon}
PRE
AI
DOI：10.1109/TNET.2024.3387948.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The surging scale of distributed training (DT) incurs significant communication overhead in datacenters, while a promising solution is in-network aggregation (INA). It leverages programmable switches (e.g., Intel Tofino switches) for gradient aggregation to accelerate DT tasks. Due to switches' limited on-chip memory size, existing solutions try to design the memory sharing mechanism for INA. This mechanism requires gradients to arrive at switches synchronously, while network dynamics make it common for the asynchronous arrival of gradients, resulting in existing solutions being inefficient (e.g., massive communication overhead). To address this issue, we propose GOAT, the first-of-its-kind work on gradient scheduling with collaborative in-network aggregation, so that switches can efficiently aggregate asynchronously arriving gradients. Specifically, GOAT first partitions the model into a set of sub-models, then decides which sub-model gradients each switch is responsible for aggregating exclusively and to which switch each worker should send its sub-model gradients. To this end, we design an efficient knapsack-based randomized rounding algorithm and formally analyze the approximation performance. We implement GOAT and evaluate its performance on a testbed consisting of 3 Intel Tofino switches and 9 servers. Experimental results show that GOAT can speed up the DT by 1.5 x compared to the state-of-the-art solutions.
Keyword:
In-network aggregation
gradient scheduling
distributed training
datacenter network
programmable network
期刊
I
IF:
3.6
论文数:
4.4K
被引数:
9.5K
机构
引用论文
分享收藏Adaptive Control of Local Updating and Model Compression for Efficient Federated Learning用于有效联邦学习的局部更新和模型压缩的自适应控制
PREAI
分享收藏分享收藏Vinyl polymers containing amido and carboxylic groups as side substituents: I. Synthesis of N-acryloyl-glycine and N-acryloyl-6-caproic acid and their grafting on cellulose membranes.png)
Polymer
PolymerIF0
PREAI
分享收藏分享收藏
查看更多内容