Self-adaptive processing graph with operator fission for elastic stream processing

doi:10.1016/j.jss.2016.06.010

2017-05-01

PRE

Nicolás Hidalgo *

Daniel Wladdimiro

Erika Rosas

DOI：10.1016/j.jss.2016.06.010

原文链接

原文求助

摘要

En 中文

Nowadays, information generated by the Internet interactions is growing exponentially, creating massive and continuous flows of events from the most diverse sources. These interactions contain valuable information for domains such as government, commerce, and banks, among others. Extracting information in near real-time from such data requires powerful processing tools to cope with the high-velocity and the high-volume stream of events. Specially designed distributed processing engines build a graph-based topology of a static number of processing operators creating bottlenecks and load balance problems when processing dynamic flows of events. In this work we propose a self-adaptive processing graph that provides elasticity and scalability by automatically increasing or decreasing the number of processing operators to improve performance and resource utilization of the system. Our solution uses a model that monitors, analyzes and changes the graph topology with a control algorithm that is both reactive and proactive to the flow of events. We have evaluated our solution with three stream processing applications and results show that our model can adapt the graph topology when receiving events at high rate with sudden peaks, producing very low costs of memory and CPU usage. (C) 2016 Elsevier Inc. All rights reserved.

Keyword:

Stream processing

Self-adaptable graph

Elastic processing

Scalable processing