.png){kind=icon}
返回
A Data Skew Oriented Reduce Placement Algorithm Based on Sampling
.png){kind=icon}
PRE
AI
DOI：10.1109/TCC.2016.2607738.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
For frequent disk I/O and large data transmissions among different racks and physical nodes, intermediate data communication has become the most important performance bottle-neck in most running Hadoop systems. This paper proposes a reduce placement algorithm called CORP to schedule related map and reduce tasks on the near nodes of clusters or racks for data locality. Because the number of keys cannot be counted until the input data are processed by map tasks, this paper applies a reservoir algorithm for sampling the input data, which can bring the distribution of keys/values closer to the overall situation of original data. Based on the distribution matrix of the intermediate results in each partition, by calculating the distance and cost matrices among the cross node communication, the related map and reduce tasks can be scheduled to relatively nearby physical nodes for data locality. We implement CORP in Hadoop 2.4.0 and evaluate its performance using three widely used benchmarks: Sort, Grep, and Join. In these experiments, an evaluation model is proposed for selecting the appropriate sample rates, which can comprehensively consider the importance of cost, effect, and variance in sampling. Experimental results show that CORP can not only improve the balance of reduces tasks effectively but also decreases the job execution time for the lower inner data communication. Compared with some other reduce scheduling algorithms, the average data transmission of the entire system on the core switch has been reduced substantially.
Keyword:
Data sampling
data skew
inner communication
MapReduce
reduce placement
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
I
IF:
5
论文数:
1.8K
被引数:
4.3K
机构
引用论文
Graph Independent Component Analysis Reveals Repertoires of Intrinsic Network Components in the Human Brain.png)
PLoS ONE
OAAI
PLoS ONEIF0
OAAI分享收藏Semen Donors as the Source of Sexually Transmitted Diseases in Artificially Inseminated Women: The Saga Unfolds精液捐赠者是人工授精妇女性传播疾病的来源: 传奇故事展开
PREAI
分享收藏分享收藏Dynamic energy efficient data placement and cluster reconfiguration algorithm for MapReduce framework面向MapReduce框架的动态节能数据放置和集群重构算法
PREAI
分享收藏
查看更多内容