.png){kind=icon}
返回
Dependency-Aware Data Locality for MapReduce
.png){kind=icon}
PRE
AI
DOI：10.1109/TCC.2015.2511765.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
MapReduce effectively partitions and distributes computation workloads to a cluster of servers, facilitating today's big data processing. Given the massive data to be dispatched, and the intermediate results to be collected and aggregated, there have been a significant studies on data locality that seeks to co-locate computation with data, so as to reduce cross-server traffic in MapReduce. They generally assume that the input data have little dependency with each other, which however is not necessarily true for that of many real-world applications, and we show strong evidence that the finishing time of MapReduce tasks can be greatly prolonged with such data dependency. In this paper, we present Dependency-Aware Locality for MapReduce (DALM) for processing the real-world input data that can be highly skewed and dependent. DALM accommodates data-dependency in a data-locality framework, organically synthesizing the key components from data reorganization, replication, placement. Beside algorithmic design within the framework, we have also closely examined the deployment challenges, particularly in public virtualized cloud environments, and have implemented DALM on Hadoop 1.2.1 with Giraph 1.0.0. Its performance has been evaluated through both simulations and real-world experiments, and compared with that of state-of-the-art solutions.
Keyword:
MapReduce
data locality
data dependency
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
I
IF:
5
论文数:
1.8K
被引数:
4.3K
机构
引用论文
LYME DISEASE: PREVALENCE AND CLINICAL IMPORTANCE OF BORRELIA BURGDORFERI SPECIFIC IgG IN FORESTRY WORKERS.png)
The Lancet
The LancetIF0
PREAI
分享收藏Investigating the effects of business process orientation on organizational innovation performance
PREAI
分享收藏
查看更多内容