.png){kind=icon}
返回
Communication Optimization Algorithms for Distributed Deep Learning Systems: A Survey
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2023.3323282.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep learning's widespread adoption in various fields has made distributed training across multiple computing nodes essential. However, frequent communication between nodes can significantly slow down training speed, creating a bottleneck in distributed training. To address this issue, researchers are focusing on communication optimization algorithms for distributed deep learning systems. In this paper, we propose a standard that systematically classifies all communication optimization algorithms based on mathematical modeling, which is not achieved by existing surveys in the field. We categorize existing works into four categories based on the optimization strategies of communication: communication masking, communication compression, communication frequency reduction, and hybrid optimization. Finally, we discuss potential future challenges and research directions in the field of communication optimization algorithms for distributed deep learning systems.
Keyword:
Communication optimization algorithms
distributed computing
distributed deep learning
parallel algorithms
期刊
IF:
6
论文数:
5.2K
被引数:
1.1W
机构
引用论文
Late Quaternary Pteropod Preservation in Eastern North Atlantic Sediments in Relation to Changing Climate.png)
Radiocarbon
OAAI
RadiocarbonIF0
OAAI分享收藏Automatic generation of fixed-point-finding evaluators for circular, but well-defined, attribute grammars自动生成用于循环但定义明确的属性语法的定点查找评估器
PREAI
分享收藏Metal to Insulator Transition in Epitaxial Graphene Induced by Molecular Doping分子掺杂诱导外延石墨烯的金属-绝缘体转变
OAAI
OAAI分享收藏
查看更多内容