.png){kind=icon}
Return
Performance Optimization of HPC Workloads in Cloud Using AI-Driven Algorithms
.png){kind=icon}
PRE
AI
DOI：10.1007/978-981-95-3585-9_13.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
As High-Performance Computing (HPC) workloads increasingly migrate to cloud infrastructures, the need for intelligent, inference-time adaptive scheduling becomes critical. Conventional schedulers such as FCFS or SJF often struggle to adapt to the heterogeneous and dynamic nature of cloud-based systems, leading to inefficient resource utilization and increased job wait times. This paper proposes a multistage AI-based pipeline to address these challenges through the integration of three core capabilities: job runtime prediction using supervised learning, anomaly detection via deep autoencoders, and adaptive resource scheduling using reinforcement learning. Leveraging real-world data from the MIT SuperCloud dataset, our system extracts meaningful patterns from time-series telemetry to support informed scheduling decisions. The job prediction module estimates runtimes based on CPU utilization, memory consumption, and I/O statistics. The anomaly detection module flags abnormal jobs using learned GPU performance norms. The RL scheduler dynamically matches jobs to compute nodes based on predicted duration and anomaly status, optimizing for turnaround time and utilization. Experimental evaluations demonstrate a 28% reduction in average turnaround time and over 10% increase in resource utilization compared to traditional schedulers. These results establish the viability of AI-driven orchestration strategies in HPC cloud platforms and underscore the importance of integrated learning-based systems in achieving scalable, efficient, and context-aware workload management.
Keywords:
Cloud HPC
Scheduling
Reinforcement Learning
Runtime Prediction
Anomaly Detection
AI Optimization
Journal
P
IF:
0
Papers:
13
Citations:
0
Organization
Cited Papers
.png){kind=icon}
Scaling TensorFlow, PyTorch, and MXNet using MVAPICH2 for High-Performance Deep Learning on Frontera
PREAI
ShareSaveShareSaveShareSaveShareSave
View more