.png){kind=icon}
.png){kind=icon}
Return
Remaining Workload-Aware Dynamic Task Scheduling Algorithm on Spark Heterogeneous Systems
.png){kind=icon}
PRE
AI
DOI：10.1109/TC.2026.3688594.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
In heterogeneous distributed computing platforms, task execution containers (e.g., Spark executors) often exhibit significant performance variations. However, most task schedulers greedily utilize resources based on the ‘first-release-first-use’policy. This leads to load imbalance across heterogeneous executors and poor application performance. To address this issue, we first construct a heterogeneous system task execution model. Then, we formalize the load-balancing task scheduling problem in heterogeneous environments as a minimum weighted executor waiting time problem and prove its NP-hardness. Next, a remaining workload-aware task scheduling strategy is proposed to address load imbalance among heterogeneous executors. Additionally, considering the differences in executor startup overhead, we introduce an earliest available executor wait mechanism to further optimize load-balancing. We comprehensively evaluate our proposed approaches using seven typical applications in two real-world heterogeneous environments. Experimental results demonstrate that our approaches reduce the standard deviation of task counts by an average of 60.6% and achieve a performance improvement of up to 35.5%.
Keywords:
Task scheduling
load-balancing
heterogeneous clusters
Spark executor
Journal
IF:
3.8
Papers:
5.3K
Citations:
9.8K