.png){kind=icon}
Return
Large-scale incremental processing with MapReduce
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2013.09.010.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
An important property of today's big data processing is that the same computation is often repeated on datasets evolving over time, such as web and social network data. While repeating full computation of the entire datasets is feasible with distributed computing frameworks such as Hadoop, it is obviously inefficient and wastes resources. In this paper, we present HadUP (Hadoop with Update Processing), a modified Hadoop architecture tailored to large-scale incremental processing with conventional MapReduce algorithms. Several approaches have been proposed to achieve a similar goal using task-level memoization. However, task-level memoization detects the change of datasets at a coarse-grained level, which often makes such approaches ineffective. Instead, HadUP detects and computes the change of datasets at a fine-grained level using a deduplication-based snapshot differential algorithm (D-SD) and update propagation. As a result, it provides high performance, especially in an environment where task-level memoization has no benefit. HadUP requires only a small amount of extra programming cost because it can reuse the code for the map and reduce functions of Hadoop. Therefore, the development of HadUP applications is quite easy. (C) 2013 Elsevier B.V. All rights reserved.
Keywords:
Big data processing
Incremental processing
MapReduce
Hadoop
Data deduplication
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
F
IF:
6.1
Papers:
6.9K
Citations:
2.3W
Organization
Cited Papers
Classifying hand sensorimotor functions of the chronic kidney disease patients using novel manual tactile test and pinch-holding-up activity.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAIShareSaveMetal carbonyls—II Carbon monoxide and chloride exchange with some cyclopentadienyl metal carbonyls
PREAI
ShareSave
View more