.png){kind=icon}
.png){kind=icon}
Return
Cost-efficient and topology-aware scheduling algorithms in distributed stream computing systems
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2025.108340.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
With the rapid growth of data volume and increasing real-time processing requirements, stream processing systems face challenges of execution inefficiency and excessive resource consumption. Apache Storm employs a simplistic round-robin scheduling strategy by default, neglecting node heterogeneity, task topology, and varying traffic patterns, leading to performance degradation and resource wastage. To address these limitations, this paper proposes two novel scheduling strategies: a resource-cost and topology-aware distributed method (MMO-Stream) and a resource-aware cooperative strategy (D-Storm). MMO-Stream integrates a cost-effective Quality-of-Service (QoS) model with a meta-heuristic-based multi-criteria optimization algorithm to optimize resource consumption, latency, and throughput simultaneously. D-Storm utilizes historical performance data and resource-awareness mechanisms to dynamically optimize task reallocation, mitigating performance deterioration from frequent rescheduling. Experimental results show MMO-Stream achieves cost-effective QoS (C-QoS) improvements of 41.7% and 39.5%, and latency reductions of 23.9% and 15.8%, compared to Storm's default scheduling and Ts-Stream, respectively. D-Storm reduces latency by 23.9% and 37.5% compared to default and Ts-Stream strategies, significantly outperforming MMO-Stream. The proposed methods effectively enhance Storm's scheduling performance and resource efficiency.
Keywords:
Stream processing
Task scheduling
Topology-aware scheduling
Resource-aware cooperative scheduling
Multi-objective optimization
Journal
F
IF:
6.1
Papers:
6.8K
Citations:
2.3W