.png){kind=icon}
返回
Optimizing Multi-Level Checkpointing for Distributed Deep Learning Workloads on Cloud Spot VM Clusters
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2024.3446770.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Spot Virtual Machines (Spot VMs) offer access to underutilized computing resources at significant discounts, sometimes up to 90% off regular on-demand pricing. For budget-conscious organizations, using clusters of Spot VMs is an effective strategy for training large-scale distributed deep learning (DDL) models. However, the risk of preemption by cloud providers poses a challenge, as it can result in the loss of unsaved data in memory and local storage. To mitigate this risk, one solution involves using networked storage systems for checkpoints, though their low write throughput can slow down training. An alternative approach is to use the memory of a remote, on-demand computing node for temporary checkpoint storage, balancing data protection with training efficiency. In this paper, we propose a novel approach, ACUTE, to optimize temporary checkpointing in the memory of on-demand nodes during DDL training. ACUTE includes three key optimizations: 1) Check-Mem, which reduces memory copying overhead on the training node; 2) Check-Trans, which accelerates checkpoint data transfer through parallel processing; and 3) Check-Pack, which eliminates unnecessary data unpacking and repacking. Implemented using PyTorch's distributed data-parallel library, ACUTE was evaluated against two other checkpointing schemes on AWS VM instances. Results show that ACUTE reduces makespan delay to nearly zero and achieves, on average, 43.30% faster checkpointing compared to a baseline multi-level checkpointing scheme, without compromising the precision of Deep Neural Network (DNN) models.
Keyword:
Checkpointing
Artificial neural networks
Parallel processing
Deep learning
Cloud computing
Graphics processing units
Fault tolerance
Distributed deep learning
cloud computing
fault-tolerant systems
checkpoint and restart
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
A Comprehensive Study of Load Balancing Approaches in the Cloud Computing Environment and a Novel Fault Tolerance Approach.png)
IEEE ACCESS
OAAI
IEEE ACCESSIF3.6
OAAI分享收藏RANKL expression is related to treatment outcome of patients with localized, high‐grade osteosarcoma
PREAI
分享收藏Post-failure recovery of MPI communication capability: Design and rationaleMPI通信能力的故障后恢复: 设计和原理
PREAI
分享收藏Myelin-associated glycoprotein: a role in myelination and in the inhibition of axonal regeneration?
PREAI
分享收藏
查看更多内容