.png){kind=icon}
返回
Improving MapReduce Performance with Partial Speculative Execution
.png){kind=icon}
PRE
AI
DOI：10.1007/s10723-015-9350-y.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The MapReduce framework has become the de facto standard for big data processing due to its attractive features and abilities. One is that it automatically parallelizes a job into multiple tasks and transparently handles task execution on a large cluster of commodity machines. The increasing heterogeneity of distributed environments may result in a few straggling tasks, which prolong job completion. Speculative execution is proposed to mitigate stragglers. However, the existing speculative execution mechanism could not work efficiently as many speculative tasks are still slower than their original tasks. In this paper, we explore an approach to increase the efficiency of speculative execution, and further improve MapReduce performance. We propose the Partial Speculative Execution (PSE) strategy to make speculative tasks start from the checkpoint. By leveraging the checkpoint of original tasks, PSE can eliminate the costs of re-reading, re-copying, and re-computing the processed data. We implement PSE in Hadoop, and evaluate its performance in terms of job completion time and the efficiency of speculative execution under several kinds of classical workloads. Experimental results show that, in heterogeneous environments with stragglers, PSE completes jobs 56 % faster than that with no speculation and 12 % faster than that with LATE, an improved speculative execution algorithm. In addition, on average PSE can improve the efficiency of speculative execution by 24 % compared to LATE.
Keyword:
Speculative execution
MapReduce performance
Straggler mitigation
期刊
IF:
2.9
论文数:
762
被引数:
1.2K
机构
引用论文
Identification of two immortalized cell lines, ECV304 and bEnd3, for in vitro permeability studies of blood-brain barrier鉴定两种永生化细胞系ECV304和bEnd3，用于血脑屏障的体外通透性研究.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏SHadoop: Improving MapReduce performance by optimizing job execution mechanism in Hadoop clusters
PREAI
分享收藏
查看更多内容