.png){kind=icon}
返回
Accelerating Distributed DNN Training via Transport Layer Scheduling
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2023.3250462.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Communication scheduling is crucial to accelerate the training of large deep learning models, in which the transmission order of layer-wise deep neural network (DNN) tensors is determined for a better computation-communication overlap. Prior approaches adopt user-level tensor partitioning to enhance the priority scheduling with finer granularity. However, a startup time slot inserted before every tensor partition will neutralize this scheduling gain. Tuning hyper-parameters for tensor partitioning is difficult, especially when the network bandwidth is shared or time-varying in multi-tenant clusters. In this article, we propose Mercury, a simple transport layer scheduler that moves the priority scheduling to the transport layer at the packet granularity. The packets with the highest priority in the Mercury buffer will be transmitted first. Mercury achieves the near-optimal overlapping between communication and computation. It also leverages the immediate aggregation at the transport layer to enable the full overlapping of gradient push and pull. We implement Mercury in MXNet and conduct comprehensive experiments on five popular DNN models in various environments. Mercury can well adapt to dynamic communication and computation resources. Experiments show that Mercury accelerates the training by up to 130% compared to the classical PS architecture, and 104% compared to state-of-the-art tensor partitioning methods.
Keyword:
Tensors
Training
Processor scheduling
Computer architecture
Computational modeling
Bandwidth
Servers
Computation-communication overlap
distri- buted machine learning
parameter server
transport layer scheduling
期刊
IF:
6
论文数:
5.2K
被引数:
1.1W
机构
引用论文
.png){kind=icon}
Does history repeat itself? Economic development and policy convergence in Vietnam and South Korea
PREAI
分享收藏Effects of Perinatal Progesterone on Mouse Mammary Differentiation In Vitro:
Brief Communication
2
3
PREAI
分享收藏分享收藏
查看更多内容