Large-scale incremental processing with MapReduce

Daewoo Lee

doi:10.1016/j.future.2013.09.010

Large-scale incremental processing with MapReduce

2014-07-01

29

PRE

AI

DOI：10.1016/j.future.2013.09.010

原文链接

原文求助

摘要

En 中文

An important property of today's big data processing is that the same computation is often repeated on datasets evolving over time, such as web and social network data. While repeating full computation of the entire datasets is feasible with distributed computing frameworks such as Hadoop, it is obviously inefficient and wastes resources. In this paper, we present HadUP (Hadoop with Update Processing), a modified Hadoop architecture tailored to large-scale incremental processing with conventional MapReduce algorithms. Several approaches have been proposed to achieve a similar goal using task-level memoization. However, task-level memoization detects the change of datasets at a coarse-grained level, which often makes such approaches ineffective. Instead, HadUP detects and computes the change of datasets at a fine-grained level using a deduplication-based snapshot differential algorithm (D-SD) and update propagation. As a result, it provides high performance, especially in an environment where task-level memoization has no benefit. HadUP requires only a small amount of extra programming cost because it can reuse the code for the map and reduce functions of Hadoop. Therefore, the development of HadUP applications is quite easy. (C) 2013 Elsevier B.V. All rights reserved.

Keyword:

Big data processing

Incremental processing

MapReduce

Hadoop

Data deduplication

Large-scale incremental processing with MapReduce

摘要

AI总结

期刊

机构