.png){kind=icon}
.png){kind=icon}
Return
VAYU: Accelerating stream processing applications through dynamic network-aware topology re-optimization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.jpdc.2017.06.023.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Stream processing applications for online analytics are commonly used in domains ranging from sensor data processing to social networking. To achieve high-throughput, stream processing engines support pipelined execution, low-overhead fault-tolerance, and efficient group communication overlays. The throughput of pipelined application workflows is significantly impacted by dynamic system state. In particular, we show that a single bottleneck in the pipeline (congested link or an overloaded operator) can drastically impact the system throughput. In this paper, we present a number of techniques for addressing bottlenecks in stream engines. Our techniques fall into two major classes - network-aware routing for fine grained control of streams; and dynamic overlay generation for optimizing performance of group communication operations. To enable fast workflow re-optimization, we present a light-weight protocol for consistent modification of pipelines. We present detailed algorithms, their implementation in a real system, and address issues of fault tolerance and performance. We evaluate performance of the proposed techniques in the context of three real applications. We show that our techniques improve performance by 20% to 200%, under various overheads, relative to a baseline representative of current implementations. We demonstrate that our techniques are robust to highly dynamic state, as well as complex congestion patterns. Given the widespread use of streaming systems and the need for dealing with dynamic system state, our techniques represent a significant and practical improvement. (C) 2017 Elsevier Inc. All rights reserved.
Keywords:
Stream-processing
Dynamic Reoptimization
Storm
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
4
Papers:
3.8K
Citations:
4.8K
