A framework for reliable and efficient data placement in distributed computing systems

doi:10.1016/j.jpdc.2005.04.019

A framework for reliable and efficient data placement in distributed computing systems

2005-10-01

71

PRE

AI

DOI：10.1016/j.jpdc.2005.04.019

Original

Original request for help

Save

Abstract

En 中文

Data placement is an essential part of today's distributed applications since moving the data close to the application has many benefits. The increasing data requirements of both scientific and commercial applications, and collaborative access to these data make it even more important. In the current approach, data placement is regarded as a side affect of computation. Our goal is to make data placement a first class citizen in distributed computing systems just like the computational jobs. They will be queued, scheduled, monitored, managed, and even checkpointed. Since data placement jobs have different characteristics than computational jobs, they cannot be treated in the exact same way as computational jobs. For this purpose, we are proposing a framework which can be considered as a data placement subsystem for distributed computing systems, similar to the I/O subsystem in operating systems. This framework includes a specialized scheduler for data placement, a high level planner aware of data placement jobs, a resource broker/policy enforcer and some optimization tools. Our system can perform reliable and efficient data placement, it can recover from all kinds of failures without any human intervention, and it can dynamically adapt to the environment at the execution time. (c) 2005 Elsevier Inc.

Keywords:

distributed computing

reliable and efficient data placement

scheduling run-time adaptation

protocol auto-tuning

data-intensive applications

I/O subsystem

A framework for reliable and efficient data placement in distributed computing systems

Abstract

AI Summary

Journal

Organization