.png){kind=icon}
.png){kind=icon}
Return
Revisiting workflow execution in HPC: a data-flow approach
.png){kind=icon}
PRE
AI
DOI：10.1007/s42514-025-00256-9.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Scientific workflows are essential to modern scientific computing, yet traditional execution approaches-based on control-flow paradigms and disk-based data transfers-struggle as data movement, rather than computation, emerges as the dominant performance bottleneck. These methods suffer from long latency due to centralized orchestration, sequential task triggering, and inefficient disk-mediated exchanges. We propose HPCFlow, a data-flow-oriented workflow framework designed for high-performance computing (HPC) environments. HPCFlow supports decentralized, input-driven execution. Functions are decomposed into computation and data transmission, enabling asynchronous data propagation and efficient overlap. HPCFlow incorporates context-aware data transfer strategies and alleviates small-file I/O inefficiencies through mini-batching. Additionally, HPCFlow implements an input synchronization mechanism to guarantee data completeness during parallel execution under elastic scaling conditions. Empirical results from a production HPC environment demonstrate that compared to a control-flow baseline, HPCFlow significantly reduces makespan and end-to-end latency, achieves efficient overlap, and alleviates pressure on network file systems, thereby validating its effectiveness for data-intensive scientific workflows.
Keywords:
Scientific Workflows
High-Performance Computing (HPC)
Cloud Computing
Data-Flow Execution
Data Movement Optimization
Journal
C
IF:
1.9
Papers:
38
Citations:
253