.png){kind=icon}
返回
Communication-efficient ADMM-based distributed algorithms for sparse training
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2023.126456.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In large-scale distributed machine learning (DML), the synchronization efficiency of the distributed algorithm becomes a critical factor that affects the training time of machine learning models as the computing scale increases. To address this challenge, we propose a novel algorithm called Grouped Sparse AllReduce based on the 2D-Torus topology (2D-TGSA), which enables constant transmission traffic that does not change with the number of workers. Our experimental results demonstrate that 2D-TGSA outperforms several benchmark algorithms in terms of synchronization efficiency. Moreover, we integrate the general form consistent ADMM with 2D-TGSA to develop a distributed algorithm (2D-TGSAADMM) that exhibits excellent scalability and can effectively handle large-scale distributed optimization problems. Furthermore, we enhance 2D-TGSA-ADMM by adopting the resilient adaptive penalty parameter approach, resulting in a new algorithm called 2D-TGSA-TPADMM. Our experiments on training the logistic regression model with '1-norm on the Tianhe-2 supercomputing platform demonstrate that our proposed algorithm can significantly reduce the synchronization time and training time compared to state-of-the-art methods.& COPY; 2023 Elsevier B.V. All rights reserved.
Keyword:
ADMM
Grouped Sparse AllReduce
Two-dimensional torus topology
Synchronization algorithm
期刊
IF:
6.5
论文数:
2.5W
被引数:
6.5W
机构
引用论文
.png){kind=icon}
DQC-ADMM: Decentralized Dynamic ADMM With Quantized and Censored CommunicationsDqc-admm: 具有量化和审查通信的分散式动态ADMM
PREAI
分享收藏Machine learning in medical applications: A review of state-of-the-art methods医学应用中的机器学习: 最新方法的回顾
PREAI
分享收藏分享收藏
查看更多内容