.png){kind=icon}
Return
A self-adaptive scheduling algorithm for reduce start time
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2014.08.011.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
MapReduce is by far one of the most successful realizations of large-scale data-intensive cloud computing platforms. When to start the reduce tasks is one of the key problems to advance the MapReduce performance. The existing implementations may result in a block of reduce tasks. When the output of map tasks become large, the performance of a MapReduce scheduling algorithm will be influenced seriously. Through analysis for the current MapReduce scheduling mechanism, this paper illustrates the reasons of system slot resources waste, which results in the reduce tasks waiting around, and proposes an optimal reduce scheduling policy called SARS (Self Adaptive Reduce Scheduling) for reduce tasks' start times in the Hadoop platform. It can decide the start time point of each reduce task dynamically according to each job context, including the task completion time and the size of map output. Through estimating job completion time, reduce completion time, and system average response time, the experimental results illustrate that, when comparing with other algorithms, the reduce completion time is decreased sharply. It is also proved that the average response time is decreased by 11% to 29%, when the SARS algorithm is applied to the traditional job scheduling algorithms FIFO, FairScheduler, and CapacityScheduler. (C) 2014 Elsevier B.V. All rights reserved.
Keywords:
Big data
Hadoop
MapReduce
Reduce
Self-adaptive
Task scheduling
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
F
IF:
6.1
Papers:
6.8K
Citations:
2.3W
Organization
Cited Papers
ShareSaveShareSaveShareSaveActive and passive surveillance of yellow fever vaccine 17D or 17DD-associated serious adverse events: Systematic review.png)
Vaccine
VaccineIF0
PREAI
ShareSaveShareSaveShareSave
View more