.png){kind=icon}
.png){kind=icon}
Return
Efficient Cross-Cloud Partial Reduce With CREW
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2024.3460185.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
By allowing p out of n workers to conduct all reduce operations among them for a round of synchronization, partial reduce, a promising partially-asynchronous variant of all reduce, has shown its power in alleviating the impacts of stragglers for iterative distributed machine learning (DML). Current partial reduce solutions are mainly designed for intra-cluster DML, in which workers are networked with high-bandwidth LAN links. Yet no prior work has looked into the problem of how to achieve efficient partial reduce for cross-cloud DML, where inter-worker connections are with scarcely-available capacities. To fill the gap, in this paper, we propose CREW, a flexible and efficient implementation of partial reduce for cross-cloud DML. At the high level, CREW is built upon the novel design of employing all active workers along with their internal connection capacities to execute the involved communication and computation tasks; and at the low level, CREW employs a suite of algorithms to distribute the tasks among workers in a load-balanced way, and deal with possible outages of workers/connections, and bandwidth contention. Detailed performance studies confirm that, CREW not only shortens the execution of each partial reduce operation, outperforming existing communication schemes such as PS, Ring, TopoAdopt, and BLINK greatly, but also significantly accelerates the training of large models, up to 15x and 9x, respectively, when compared with the all-to-all direct communication scheme and original partial reduce design.
Keywords:
Training
Wide area networks
Synchronization
Cloud computing
Machine learning
Data models
Costs
Partial reduce
flow scheduling
cloud computing
Journal
IF:
6
Papers:
5.2K
Citations:
1.1W