.png){kind=icon}
返回
SDCC: software-defined collective communication for distributed training
.png){kind=icon}
PRE
AI
DOI：10.1007/s11432-023-3894-4.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Communication is crucial to the performance of distributed training. Today's solutions tightly couple the control and data planes and lack flexibility, generality, and performance. In this study, we present SDCC, a software-defined collective communication framework for distributed training. SDCC is based on the principle of modern systems design to effectively decouple the control plane from the data plane. SDCC abstracts the operations for collective communication in distributed training with dataflow operations and unifies computing and communication with a single dataflow graph. The abstraction, together with the unification, is powerful: it enables users to easily express new and existing collective communication algorithms and optimizations, simplifies the integration with different computing engines (e.g., PyTorch and TensorFlow) and network transports (e.g., Linux TCP and kernel bypass), and allows the system to improve performance by exploiting parallelism exposed by the dataflow graph. We further demonstrate the benefits of SDCC in four use cases.
Keyword:
machine learning systems
distributed training
deep learning
collective communication
software-defined networking
期刊
IF:
7.6
论文数:
4.9K
被引数:
8.9K
机构
引用论文
.png){kind=icon}
An autonomous 16 mm/sup 3/ solar-powered node for distributed wireless sensor networks用于分布式无线传感器网络的自治16毫米/sup3/太阳能节点
PREAI
分享收藏
Does history repeat itself? Economic development and policy convergence in Vietnam and South Korea
PREAI
分享收藏分享收藏Reveal training performance mystery between TensorFlow and PyTorch in the single GPU environment揭示TensorFlow和PyTorch在单GPU环境中的训练性能之谜
PREAI
分享收藏Effects of Perinatal Progesterone on Mouse Mammary Differentiation In Vitro:
Brief Communication
2
3
PREAI
分享收藏
查看更多内容