.png){kind=icon}
.png){kind=icon}
Return
Efficient Inter-Datacenter AllReduce With Multiple Trees
.png){kind=icon}
PRE
AI
DOI：10.1109/TNSE.2024.3419030.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In this paper, we look into the problem of achieving efficient inter-datacenter AllReduce operations for geo-distributed machine learning (Geo-DML). Compared with intra-datacenter distributed training, the heterogeneous wide-area network (WAN) connections among Geo-DML workers are scarce, expensive, and unstable, making existing proposals designed for homogeneous networks fall short. Despite that some recent optimizations have been proposed for Geo-DML, they break the consistency semantics of bulk synchronous parallel (BSP), thus bringing no benefit to the widely existing BSP-based applications. To address these issues, we propose mTree, a topology management suite for Geo-DML. With the global view of the heterogeneous WAN connections, mTree builds multiple optimized spanning trees along with suggested workload distribution proportions, respecting the constraints of both the number of trees and their maximum height specified by the training. Based on these results, geo-distributed workers could launch concurrent tree-based pipelined AllReduce operations to make efficient use of the heterogeneous network. Detailed performance studies on real-world network topologies imply that mTree achieves efficient AllReduce, significantly outperforming existing solutions.
Keywords:
Training
Vegetation
Synchronization
Data models
Wide area networks
Tensors
Peer-to-peer computing
AllReduce
communication optimization
inter-DC WAN
Journal
I
IF:
7.9
Papers:
2.5K
Citations:
10.0K