.png){kind=icon}
返回
Handling data skew in join algorithms using MapReduce
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2015.12.024.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
One of the major obstacles hindering effective join processing on MapReduce is data skew. Since MapReduce's basic hash-based partitioning method cannot solve the problem properly, two alternatives have been proposed: range-based and randomized methods. However, they still remain some drawbacks: the range-based method does not handle join product skew, and the randomized method performs worse than the basic hash-based partitioning when input relations are not skewed. In this paper, we present a new skew handling method, called multi-dimensional range partitioning (MDRP). The proposed method overcomes the limitations of traditional algorithms in two ways: 1) the number of output records expected at each, machine is considered, which leads to better handling of join product skew, and 2) a small number of input records are sampled before the actual join begins so that an efficient execution plan considering the degree of data skew can be created. As a result, in a scalar skew experiment, the proposed join algorithm is about 6.76 times faster than the range-based algorithm when join product skew exists and about 5.14 times than the randomized algorithm when input relations are not skewed. Moreover, through the worst-case analysis, we show that the input and the output imbalances are less than or equal to 2. The proposed algorithm does not require any modification to the original MapReduce environment and is applicable to complex join operations such as theta joins and multi-way joins. (C) 2016 Elsevier Ltd. All rights reserved.
Keyword:
MapReduce
Join algorithm
Skew handling
Multi-dimensional range partitioning
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
7.5
论文数:
2.9W
被引数:
10.2W
机构
引用论文
.png){kind=icon}
Reaction of Carbon Monosulfide and Monoselenide with Chalcogens and Halogens一硫化碳和单硒化物与硫属元素和卤素的反应
PREAI
分享收藏Physics of aeration in slug: flow visualization analysis in horizontal pipes段塞中曝气的物理学: 水平管中的流动可视化分析
PREAI
分享收藏
查看更多内容