.png){kind=icon}
返回
Stream-aware indexing for distributed inequality join processing
.png){kind=icon}
PRE
AI
DOI：10.1016/j.is.2024.102425.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Inequality join is an operator to join data on inequality conditions and it is a fundamental building block for applications. While methods and optimizations exist for efficient inequality join in batch processing, little attention has been given to its streaming version, particularly to large-scale data-intensive applications that run on Distributed Stream Processing Systems (DSPSs). Designing an inequality join in streaming and distributed settings is not an easy task: (i) indexes have to be employed to efficiently support inequality-based comparisons, but the continuous stream of data imposes continuous insertions, updates, and deletions of elements in the indexes-hence a huge overhead for the DSPSs; (ii) oftentimes real data is skewed, which makes indexing even more challenging. To address these challenges, we propose the Stream-Aware inequality join (STA), an indexing method that can reduce redundancy and index update overhead. STA builds a separate in-memory index structure for hotkeys, i.e., the most frequently used keys, which are automatically identified with an efficient data sketch. On the other hand, the cold keys are treated using a linked set of index structures. In this way, STA avoids many superfluous index updates for frequent items. Finally, we implement four state-of-the-art inequality join solutions for a widely employed DSPS (Apache Storm) and compare their performance with STA on four realworld data sets and a synthetic one. The results of our experimental evaluation reveal that our stream-aware approach outperforms existing solutions.
Keyword:
Distributed stream processing system
Inequality join
B plus tree indexing
Augmented sketch
Skewed data distribution
期刊
IF:
3.9
论文数:
2.8K
被引数:
1.8K
机构
引用论文
Plasma kisspeptin levels are elevated in cord blood and present sexual dimorphism in the adult population: Relation with leptin, gonadotropins and anthropometrical data.png)
Peptides
PeptidesIF0
PREAI
分享收藏Mobilizing the Mexican Immigrant: The Role of Mexican-American Organizations动员墨西哥移民：墨西哥裔美国人组织的作用
PREAI
分享收藏
分享收藏Political Attitudes, Age, and Aging: A Cohort Analysis of Archival Data政治态度、年龄与老龄化：一项基于档案数据的队列分析
PREAI
分享收藏
查看更多内容