.png){kind=icon}
返回
Efficient Time-Evolving Stream Processing at Scale
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2019.2911495.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Time-evolving stream datasets exist ubiquitously in many real-world applications where their inherent hot keys often evolve over times. Nevertheless, few existing solutions can provide efficient load balancing on these time-evolving datasets while preserving low memory overhead. In this paper, we present a novel load balancing mechanism (named FISH), which can provide the efficient time-evolving stream processing at scale through recent hot keys identification and worker assignment. The key insight of this work is that the keys of time-evolving stream data can have a skewed distribution within the bounded distance of time interval. This enables to accurately identify the recent hot keys for the real-time load balancing within a bounded scope. We therefore propose an epoch-based recent hot key identification with specialized intra-epoch frequency counting (for maintaining low memory overhead) and inter-epoch hotness decaying (for suppressing superfluous computation). We also propose to heuristically infer the accurate information of remote workers through computation rather than communication for cost-efficient worker assignment. We have integrated our approach into Apache Storm. Our results on a cluster of 128 nodes for both synthetic and real-world stream datasets show that FISH significantly outperforms state-of-the-arts with the average and the 99th percentile latency reduction by 87.12 and 76.34 percent (versus W-Choices), and memory overhead reduction by 96.66 percent (versus Shuffle Grouping).
Keyword:
Stream processing
streaming partition
load balancing
efficiency
scalability
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6
论文数:
5.2K
被引数:
1.1W
机构
暂无机构信息
引用论文
Classifying hand sensorimotor functions of the chronic kidney disease patients using novel manual tactile test and pinch-holding-up activity使用新颖的手动触觉测试和捏举活动对慢性肾脏疾病患者的手感觉运动功能进行分类.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏
查看更多内容