.png){kind=icon}
Return
Improving MapReduce Performance with Partial Speculative Execution
.png){kind=icon}
PRE
AI
DOI：10.1007/s10723-015-9350-y.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The MapReduce framework has become the de facto standard for big data processing due to its attractive features and abilities. One is that it automatically parallelizes a job into multiple tasks and transparently handles task execution on a large cluster of commodity machines. The increasing heterogeneity of distributed environments may result in a few straggling tasks, which prolong job completion. Speculative execution is proposed to mitigate stragglers. However, the existing speculative execution mechanism could not work efficiently as many speculative tasks are still slower than their original tasks. In this paper, we explore an approach to increase the efficiency of speculative execution, and further improve MapReduce performance. We propose the Partial Speculative Execution (PSE) strategy to make speculative tasks start from the checkpoint. By leveraging the checkpoint of original tasks, PSE can eliminate the costs of re-reading, re-copying, and re-computing the processed data. We implement PSE in Hadoop, and evaluate its performance in terms of job completion time and the efficiency of speculative execution under several kinds of classical workloads. Experimental results show that, in heterogeneous environments with stragglers, PSE completes jobs 56 % faster than that with no speculation and 12 % faster than that with LATE, an improved speculative execution algorithm. In addition, on average PSE can improve the efficiency of speculative execution by 24 % compared to LATE.
Keywords:
Speculative execution
MapReduce performance
Straggler mitigation
Journal
IF:
2.9
Papers:
762
Citations:
1.2K
Organization
Cited Papers
ShareSaveIdentification of two immortalized cell lines, ECV304 and bEnd3, for in vitro permeability studies of blood-brain barrier.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAIShareSaveSHadoop: Improving MapReduce performance by optimizing job execution mechanism in Hadoop clusters
PREAI
ShareSaveShareSave
View more