.png){kind=icon}
返回
Pilot-Data: An abstraction for distributed data
.png){kind=icon}
OA
AI
DOI：10.1016/j.jpdc.2014.09.009.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Scientific problems that depend on processing large amounts of data require overcoming challenges in multiple areas: managing large-scale data distribution, controlling co-placement and scheduling of data with compute resources, and storing, transferring, and managing large volumes of data Although there exist multiple approaches to address each of these challenges and the complexity of distributed environments, an integrative approach is missing; furthermore, extending existing functionality or enabling interoperable capabilities remains difficult at best. We propose the concept of Pilot-Data to address the fundamental challenges of co-placement and scheduling of data and compute in heterogeneous and distributed environments with interoperability and extensibility as first-order concerns. Pilot-Data is an extension of the Pilot-job abstraction for supporting the management of data in conjunction with compute tasks. Pilot-Data separates logical data units from physical storage, thereby providing the basis for efficient compute/data placement and scheduling. In this paper, we discuss the design and implementation of the Pilot-Data prototype, demonstrate its use by data-intensive applications on multiple production distributed cyberinfrastructure and illustrate the advantages arising from flexible execution modes enabled by Pilot-Data. Our experiments utilize an implementation of Pilot-Data in conjunction with a scalable Pilot-job (BigJob) to establish the application performance that can be enabled by the use of Pilot-Data. We demonstrate how the concept of Pilot-Data also provides the basis upon which to build tools and support capabilities like affinity which in turn can be used for advanced data-compute co-placement and scheduling. (C) 2014 Elsevier Inc. All rights reserved.
Keyword:
Data-intensive
Distributed computing
Pilot-jobs
Grid
HTC
HPC
Cloud
Bigdata
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4
论文数:
3.8K
被引数:
4.8K
机构
引用论文
Synthesis and characterization of IMPY derivatives that regulate metal-induced amyloid-β aggregation.png)
Metallomics
OAAI
MetallomicsIF0
OAAI分享收藏分享收藏Influence of Dietary Zinc, Calcium and Vitamin D for Hens on Zinc Content of Tissues and Eggs and on Bone Composition日粮锌、钙和维生素D对产蛋鸡组织锌含量及骨骼组成的影响OAAI
OAAI分享收藏Globus Online Accelerating and Democratizing Science through Cloud-Based ServicesGlobus Online通过基于云的服务加速和民主化科学
PREAI
分享收藏
查看更多内容