.png){kind=icon}
.png){kind=icon}
Return
Communication Optimization for Large Language Model Distributed Training: A Systematic Survey
.png){kind=icon}
PRE
AI
DOI：10.1007/978-981-95-3324-4_26.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The emergence of trillion-parameter large language models(LLMs) like GPT-4 and PaLM-2 has fundamentally transformed artificial general intelligence capabilities. However, their inefficient synchronization strategy and suboptimal network resource utilization together limit the system's scalability and training efficiency. This survey provides a systematic categorization of communication optimization techniques. It introduces a hierarchical framework that differentiates between logical strategy optimization and physical resource orchestration. Strategy optimizations focus on adaptive synchronization and gradient compression to reduce communication volume. Resource optimizations improve task scheduling, network topology, and hardware adaptation to boost physical efficiency. This work classifies current optimization strategies for large-scale distributed training systems. Further, it discusses potential research directions for improving communication efficiency in next-generation LLM distributed training systems.
Keywords:
communication optimization
distributed training
distributed parallelism
large scale
Large Language Model
Journal
P
IF:
0
Papers:
44
Citations:
0