.png){kind=icon}
返回
MapReduce with communication overlap (MaRCO)
.png){kind=icon}
PRE
AI
DOI：10.1016/j.jpdc.2012.12.012.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
MapReduce is a programming model from Google for cluster-based computing in domains such as search engines, machine learning, and data mining. MapReduce provides automatic data management and fault tolerance to improve programmability of clusters. MapReduce's execution model includes an all-map-to-all-reduce communication, called the shuffle, across the network bisection. Some MapReductions move large amounts of data (e.g., as much as the input data), stressing the bisection bandwidth and introducing significant runtime overhead. Optimizing such shuffle-heavy MapReduction's is important because (1) they include key applications (e.g., inverted indexing for search engines and data clustering for machine learning) and (2) they run longer than shuffle-light MapReductions (e.g., 5x longer). In MapReduce, the asynchronous nature of the shuffle results in some overlap between the shuffle and map. Unfortunately, this overlap is insufficient in shuffle-heavy MapReductions. We propose MapReduce with communication overlap (MaRCO) to achieve nearly full overlap via the novel idea of including reduce in the overlap. While MapReduce lazily performs reduce computation only after receiving all the map data, MaRCO employs eager reduce to process partial data from some map tasks while overlapping with other map tasks' communication. MaRCO's approach of hiding the latency of the inevitably high shuffle volume of shuffle-heavy MapReductions is fundamental for achieving performance. We implement MaRCO in Hadoop's MapReduce and show that on a 128-node Amazon EC2 cluster, MaRCO achieves 23% average speed-up over Hadoop for shuffle-heavy MapReductions. (C) 2012 Elsevier Inc. All rights reserved.
Keyword:
Cloud computing
Parallel computing
MapReduce
Performance optimization
Distributed processing
Large-scale data processing
期刊
IF:
4
论文数:
3.8K
被引数:
4.8K
机构
引用论文
Manipulation Therapy Relieved Pain More Rapidly Than Acupuncture among Lateral Epicondylalgia (Tennis Elbow) Patients: A Randomized Controlled Trial with 8‐Week Follow‐Up在外侧上髁痛 (网球肘) 患者中，手法治疗比针刺更快地缓解疼痛: 一项为期8周的随机对照试验
OAAI
.png){kind=icon}
OAAI分享收藏Metal carbonyls—II Carbon monoxide and chloride exchange with some cyclopentadienyl metal carbonyls
PREAI
分享收藏Homogeneous catalytic carbonylation of nitroaromatics Part III. Discovery and in situ high pressure FTIR spectral studies of a novel binuclear ruthenium catalyst硝基芳族化合物的均相催化羰基化部分III.新型双核钌催化剂的发现和原位高压FTIR光谱研究
PREAI
分享收藏The effect of substitution of a thioether donor by a phosphine donor in thiacrown complexes of iron
PREAI
分享收藏Techno Economic and Life Cycle Assessment of 1 MW Solar Parabolic Trough System for Udaipur Zone
PREAI
分享收藏 没有更多内容
没有更多内容