.png){kind=icon}
返回
XAgg: Accelerating Heterogeneous Distributed Training Through XDP-Based Gradient Aggregation
.png){kind=icon}
PRE
AI
DOI：10.1109/TNET.2023.3339524.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
With the growth of model/dataset/system size for distributed model training in datacenters, the widely used Parameter Server (PS) architecture suffers from communication bottleneck of gradient transmission. Recent works attempt to utilize programmable switches to implement in-network gradient aggregation and alleviate communication bottlenecks on PSs. Due to the limited on-chip memory of programmable switches, gradient transmission requires strict synchronization to achieve ideal aggregation performance. However, the distributed training system is usually heterogeneous in datacenters (e.g., computation and bandwidth heterogeneity), and the gradient will reach the aggregation nodes asynchronously, thereby seriously affecting the aggregation performance. To solve the above issue, we propose XAgg, which accelerates heterogeneous gradient aggregation by deploying the eXpress Data Path (XDP) based aggregator on servers. Specifically, the abundant idle memory on servers can cache the entire gradient, so as to effectively deal with asynchronous gradient transmission in heterogeneous scenarios. Moreover, XDP can provide high-performance and low-latency gradient aggregation. We conduct microbenchmark and testbed with real-world DNN models and datasets. Experimental results show that XAgg improves the gradient aggregation throughput by 3.3x compared with TCP-based aggregation, reaching 100 Gbps with 10 CPU cores. In addition, XAgg reduces communication time by 49%-82% compared with state-of-the-art solutions.
Keyword:
In-network gradient aggregation
distributed model training
eXpress data path (XDP)
期刊
I
IF:
3.6
论文数:
4.4K
被引数:
9.5K
机构
引用论文
Supramolecular solid-state architectures constructed from 4,4′-bipyridine-N,N′-dioxide and dicyanamido tectons. Synthesis and crystal structures of [M(bpno)2{N(CN)2}2(H2O)2] (M=Co, Mn) and [Cu(bpno){N(CN)2}2(H2O)].png)
Polyhedron
PolyhedronIF0
PREAI
分享收藏Evaluation of Dyslipidaemia Using an Algorithm of Lipid Profile Measures among Newly Diagnosed Type II Diabetes Mellitus Patients: A Cross-Sectional Study at Dormaa Presbyterian Hospital, Ghana使用血脂谱指标算法评估新诊断2型糖尿病患者的血脂异常：加纳多马教友医院横断面研究Medicina
OAAI
MedicinaIF0
OAAI分享收藏
Characteristics of Co-Allocated Online Services and Batch Jobs in Internet Data Centers: A Case Study From Alibaba CloudIEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏分享收藏
查看更多内容