.png){kind=icon}
返回
A state lossless scheduling strategy in distributed stream computing systems
.png){kind=icon}
PRE
AI
DOI：10.1016/j.jnca.2022.103462.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Stateful scheduling is of critical importance for the performance of a distributed stream computing system. In such a system, inappropriate task deployment lowers the resource utilization of cluster and introduces more communication between compute nodes. Also an online adjustment to task deployment scheme suffers slow state recovery during task restart. To address these issues, we propose a state lossless scheduling strategy (Sl-Stream) to optimize the task deployment and state recovery process. This paper discusses this strategy from the following aspects: (1) A stream application model and a resource model are constructed, together with the formalization of problems including subgraph partitioning, task deployment and stateful scheduling. (2) A multi-factor topology partitioning method is proposed using a quantum particle swarm algorithm. The assignment between tasks and nodes is optimized using a bipartite graph minimum matching algorithm. (3) A hierarchical local topology migration is performed when an online scheduling is triggered, which ensures the processing sustainability of data streams. (4) A fragment loss-tolerant jerasure tool is used to divide the state data into fragments and periodically save them in upstream vertex instances, which ensures the available fragments be able to reconstruct the whole state in parallel. (5) Metrics including latency, throughput and state recovery time are evaluated in a real distributed stream computing environment. With a comprehensive evaluation of variable-rate input scenarios, the proposed Sl-Stream system provides promising improvements on throughput, latency and state recovery time compared to the existing Storm's scheduling strategies.
Keyword:
Streamcomputing
Onlinescheduling
Statemanagement
Bipartitegraph
Hierarchicalmigration
期刊
IF:
8
论文数:
3.7K
被引数:
1.1W
机构
引用论文
.png){kind=icon}
SP-ant: An ant colony optimization based operator scheduler for high performance distributed stream processing on heterogeneous clustersSP-ant: 一种基于蚁群优化的异构集群高性能分布式流处理算子调度器
PREAI
分享收藏
查看更多内容