.png){kind=icon}
.png){kind=icon}
Return
Evaluation of Distributed Asynchronous Checkpointing in High-Performance Computing
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-032-07612-0_13.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
During the training of large models, traditional checkpointing introduces significant overhead, as it requires pausing training while model states are copied from GPU memory to storage. At the same time, frequent checkpoint is needed to ensure an efficient use of resources. Efficient checkpointing is crucial for large-scale training of Artificial Intelligence (AI) models, especially on high-performance computing (HPC) systems. In this work, we evaluate distributed asynchronous checkpointing (DACP) applied on various LLM models on the Leonardo supercomputer, hosted by CINECA. By integrating asynchronous checkpointing, we enable overlapping data transfer operations with training iterations, significantly reducing training time. Our experiments span multiple LLM configurations, leveraging PyTorch DACP to optimize checkpointing frequency and minimize graphics processing unit (GPU) idle time. We evaluate this approach at scales of up to 256 GPUs using different model sizes. Results demonstrate a substantial reduction in checkpoint overhead, achieving up to a 6x improvement compared to synchronous methods. Our evaluations highlight the benefits of asynchronous checkpointing for large-scale training and provide insights into its practical deployment on HPC infrastructures.
Keywords:
Asynchronous checkpointing
Training
HPC
LLMs
Journal
H
IF:
0
Papers:
54
Citations:
0