.png){kind=icon}
.png){kind=icon}
Return
Enabling Elasticity in Scientific Workflows for High-Performance Computing Systems
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-031-99854-6_21.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Modern workflows are increasingly data and event-driven, particularly with the growing integration of AI/ML tasks. Static workflows lack the flexibility to adapt at runtime in response to dynamic, event-driven tasks or fluctuating system demands, leading to suboptimal resource utilization and reduced efficiency. Elastic workflows, which can dynamically adjust resource usage based on system availability and workload demands, can enhance computational efficiency through intelligent resource management strategies. This paper presents the design and implementation of an elastic workflow framework for high-performance computing (HPC), highlighting key challenges and design considerations. Our framework is based on an elastic Parsl workflow manager and a custom elastic resource manager, both leveraging capabilities of an implementation of the Process Management Interface for Exscale (PMIx) API. Through real-world and synthetic workflow case studies, we demonstrate improved resource utilization and reduced execution times, showcasing the benefits of elasticity in HPC workflows.
Keywords:
Dynamic Resources
Parsl
PMIx
Scientific Workflows
Journal
E
IF:
0
Papers:
28
Citations:
0