.png){kind=icon}
Return
Accelerating Distributed Training With Collaborative In-Network Aggregation
.png){kind=icon}
PRE
AI
DOI：10.1109/TNET.2024.3387948.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The surging scale of distributed training (DT) incurs significant communication overhead in datacenters, while a promising solution is in-network aggregation (INA). It leverages programmable switches (e.g., Intel Tofino switches) for gradient aggregation to accelerate DT tasks. Due to switches' limited on-chip memory size, existing solutions try to design the memory sharing mechanism for INA. This mechanism requires gradients to arrive at switches synchronously, while network dynamics make it common for the asynchronous arrival of gradients, resulting in existing solutions being inefficient (e.g., massive communication overhead). To address this issue, we propose GOAT, the first-of-its-kind work on gradient scheduling with collaborative in-network aggregation, so that switches can efficiently aggregate asynchronously arriving gradients. Specifically, GOAT first partitions the model into a set of sub-models, then decides which sub-model gradients each switch is responsible for aggregating exclusively and to which switch each worker should send its sub-model gradients. To this end, we design an efficient knapsack-based randomized rounding algorithm and formally analyze the approximation performance. We implement GOAT and evaluate its performance on a testbed consisting of 3 Intel Tofino switches and 9 servers. Experimental results show that GOAT can speed up the DT by 1.5 x compared to the state-of-the-art solutions.
Keywords:
In-network aggregation
gradient scheduling
distributed training
datacenter network
programmable network
Journal
I
IF:
3.6
Papers:
4.4K
Citations:
9.5K
Organization
View more organizations
Cited Papers
ShareSave
ShareSaveShareSaveVinyl polymers containing amido and carboxylic groups as side substituents: I. Synthesis of N-acryloyl-glycine and N-acryloyl-6-caproic acid and their grafting on cellulose membranes.png)
Polymer
PolymerIF0
PREAI
ShareSaveShareSaveShareSaveShareSave View more