.png){kind=icon}
返回
CloudFlow: A data-aware programming model for cloud workflow applications on modern HPC systems
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2014.10.028.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Traditional High-Performance Computing (HPC) based big-data applications are usually constrained by having to move large amount of data to compute facilities for real-time processing purpose. Modern HPC systems, represented by High-Throughput Computing (HTC) and Many-Task Computing (MTC) platforms, on the other hand, intend to achieve the long-held dream of moving compute to data instead. This kind of data-aware scheduling, typically represented by Hadoop MapReduce, has been successfully implemented in its Map Phase, whereby each Map Task is sent out to the compute node where the corresponding input data chunk is located. However, Hadoop MapReduce limits itself to a one-map-to-one-reduce framework, leading to difficulties for handling complex logics, such as pipelines or workflows. Meanwhile, it lacks built-in support and optimization when the input datasets are shared among multiple applications and/or jobs. The performance can be improved significantly when the knowledge of the shared and frequently accessed data is taken into scheduling decisions. To enhance the capability of managing workflow in modern HPC system, this paper presents CloudFlow, a Hadoop MapReduce based programming model for cloud workflow applications. CloudFlow is built on top of MapReduce, which is proposed not only being data aware, but also shared-data aware. It identifies the most frequently shared data, from both task-level and job-level, replicates them to each compute node for data locality purposes. It also supports user-defined multiple Map- and Reduce functions, allowing users to orchestrate the required data-flow logic. Mathematically, we prove the correctness of the whole scheduling framework by performing theoretical analysis. Further more, experimental evaluation also shows that the execution runtime speedup exceeds 4X compared to traditional MapReduce implementation with a manageable time overhead. (C) 2014 Elsevier B.V. All rights reserved.
Keyword:
Concurrency
Data aware
MapReduce
HPC
Programming model
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
F
IF:
6.1
论文数:
6.8K
被引数:
2.3W
机构
引用论文
Classifying hand sensorimotor functions of the chronic kidney disease patients using novel manual tactile test and pinch-holding-up activity使用新颖的手动触觉测试和捏举活动对慢性肾脏疾病患者的手感觉运动功能进行分类.png)
PLOS ONE
OAAI
PLOS ONEIF0
OAAI分享收藏
分享收藏Is it wise to compete or to collaborate? Remanufacturing business models under collective extended producer responsibility legislation竞争还是合作是明智之举？集体延伸生产者责任立法下的再制造商业模式
PREAI
分享收藏Metal carbonyls—II Carbon monoxide and chloride exchange with some cyclopentadienyl metal carbonyls
PREAI
分享收藏
查看更多内容