An Improved Algorithm for Optimizing MapReduce Based on Locality and Overlapping

Jianjiang Li

doi:10.26599/TST.2018.9010115

An Improved Algorithm for Optimizing MapReduce Based on Locality and Overlapping

2018-12-01

9

OA

AI

DOI：10.26599/TST.2018.9010115

原文链接

查看原文

摘要

En 中文

MapReduce is currently the most popular programming model for big data processing, and Hadoop is a well-known MapReduce implementation platform. However, Hadoop jobs suffer from imbalanced workloads during the reduce phase and inefficiently utilize the available computing and network resources. In some cases, these problems lead to serious performance degradation in MapReduce jobs. To resolve these problems, in this paper, we propose two algorithms, the Locality-Based Balanced Schedule (LBBS) and Overlapping-Based Resource Utilization (OBRU), that optimize the Locality-Enhanced Load Balance (LELB) and the Map, Local reduce, Shuffle, and final Reduce (MLSR) phases. The LBBS collects partition information from input data during the map phase and generates balanced schedule plans for the reduce phase. OBRU is responsible for using computing and network resources efficiently by overlapping the local reduce, shuffle, and final reduce phases. Experimental results show that the LBBS and OBRU algorithms yield significant improvements in load balancing. When LBBS and OBRU are applied, job performance increases by 15% from that of models using LELB and MLSR.

Keyword:

MapReduce

overlapping

load balance

data locality

An Improved Algorithm for Optimizing MapReduce Based on Locality and Overlapping

摘要

AI总结

期刊

机构

引用论文