.png){kind=icon}
返回
Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2023.3251997.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Today, large-scale cloud organizations are deploying datacenters and edge clusters globally to provide low-latency access to services. Running stream applications across geo-distributed sites are emerging as a daily requirement. However, existing efforts have dominantly centered around stateless stream processing, leaving another urgent trend-stateful stream processing-much less explored. A driving need is to store and update states during processing, and most importantly, successfully recover large distributed states when faults and failures happen. Existing studies exhibit major limitations including: (1) they mostly inherit MapReduce's single master/many workers architecture, where the central master can easily become ascalability bottleneck; (2) they offer state recovery mainly through three approaches: replication recovery, checkpointing recovery, and DStream-based lineage recovery, which are either slow, resource-expensive or failing to handle multiple failures; and (3) they are not adaptive to heterogeneous hardware settings. We present A-FP4S, a novel adaptive fragments-based parallel state recovery mechanism for stream processing systems. A-FP4S organizes stream operators into a distributed hash table based peer-to-peer overlay and divides each node's local state into many fragments. These fragments are periodically stored in node's multiple neighbors, ensuring different sets of available fragments can reconstruct failed states in parallel. This mechanism is extremely scalable to the lost state, significantly reduces failure recovery time, and can tolerate multiple node failures. A-FP4S is adaptive to heterogeneous hardware settings by automatic parameter tuning over phases. Compared to Apache Storm, A-FP4S achieves 31.8% to 50.5% reduction in recovery latency. Large-scale experiments using real-world datasets demonstrate A-FP4S's attractive scalability and adaptivity properties.
Keyword:
Distributed hash tables
state recovery
stream processing
期刊
IF:
6
论文数:
5.2K
被引数:
1.1W
机构
引用论文
Classifying hand sensorimotor functions of the chronic kidney disease patients using novel manual tactile test and pinch-holding-up activity使用新颖的手动触觉测试和捏举活动对慢性肾脏疾病患者的手感觉运动功能进行分类.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏分享收藏分享收藏分享收藏
查看更多内容