arrow
返回

Priority-Driven Flow Scheduling for Distributed Large Language Model Training

delete2026-05-01
delete0
PRE
AI
T
Tianshi Wang
Y
Yiran Zhang *
Z
Zhang, Yiao
Z
Zhang, Qiyang
Z
Zhou, Ao
S
Shangguang Wang
DOI:10.23919/cje.2025.00.349delete
delete原文链接
delete分享
delete收藏
摘要

摘要

En 中文
大规模语言模型(LLM)训练高度依赖于分布式加速器之间的高效通信协调。现有方法主要关注独立优化特定的并行策略,但缺乏对不同通信模式的系统性优先级排序,导致训练性能次优。本文提出了一种混合并行优先级分配框架HyPA。HyPA采用离线代理建模进行闭式优先级优化,并结合在线参数感知实现动态环境适应,从而在不修改网络基础设施的情况下实现自适应带宽分配和拥塞控制。通过对真实训练工作负载的全面评估,HyPA在密集型和稀疏型LLM模型的任务完成时间上均实现了显著改进。在微基准测试中,任务完成时间最多缩短18.59%,在大型训练部署中最多缩短16%。
Keyword:
Modeling
Training
Parallel processing
Large language models
Fluid flow
Optimization
Schedules
Scheduling
Calculators
Servers
Distributed training
Flow scheduling
Pipeline parallelism
Data parallelism
Communication optimization

期刊

C
Chinese Journal of Electronics
IF:
3
论文数:
97
被引数:
1.8K

机构

北京邮电大学 封面图
北京邮电大学
学者数:
0
论文数: 193
被引数: 8.4K
北京大学 封面图
北京大学
学者数:
125
论文数: 48
被引数: 5
引用论文

引用论文

Congestion Control for Large-Scale RDMA Deployments大规模RDMA部署的拥塞控制
err2015-08-17
err0
PREAI
errYibo Zhu; Haggai Eran; Daniel Firestone; Chuanxiong Guo; Marina Lipshteyn; Yehonatan Liron; Jitendra Padhye; Shachar Raindel; Mohamad Haj Yahia; Ming Zhang
err分享
err收藏
Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
err
IF0
err2024-11-17
err0
PREAI
errWei An; Xiao Bi; Guanting Chen; Shanhuang Chen; Chengqi Deng; Honghui Ding; Kai Dong; Qiushi Du; Wenjun Gao; Kang Guan; Jianzhong Guo; Yongqiang Guo; Zhe Fu; Ying He; Panpan Huang; Jiashi Li; Wenfeng Liang; Xiaodong Liu; Xin Liu; Yiyuan Liu; Yuxuan Liu; Shanghao Lu; Xuan Lu; Xiaotao Nie; Tian Pei; Junjie Qiu; Hui Qu; Zehui Ren; Zhangli Sha; Xuecheng Su; Xiaowen Sun; Yixuan Tan; Minghui Tang; Shiyu Wang; Yaohui Wang; Yongji Wang; Ziwei Xie; Yiliang Xiong; Yanhong Xu; Shengfeng Ye; Shuiping Yu; Yukun Zha; Liyue Zhang; Haowei Zhang; Mingchuan Zhang; Wentao Zhang; Yichao Zhang; Chenggang Zhao; Yao Zhao; Shangyan Zhou; Shunfeng Zhou; Yuheng Zou
err分享
err收藏
Crux: GPU-Efficient Communication Scheduling for Deep Learning Training
err2024-08-04
err0
PREAI
errCao,Jiamin; Guan,Yu; Qian,Kun; Gao,Jiaqi; Xiao,Wencong; Dong,Jianbo; Fu,Binzhang; Cai,Dennis; Zhai,Ennan
err分享
err收藏
err分享
err收藏
PipeDream白日梦
err2019-10-27
err0
PREAI
errDeepak Narayanan; Aaron Harlap; Amar Phanishayee; Vivek Seshadri; Nikhil R. Devanur; Gregory R. Ganger; Phillip B. Gibbons; Matei Zaharia
err分享
err收藏
没有更多内容