Native data representation: An efficient wire format for high-performance distributed computing

Greg Eisenhauer

doi:10.1109/TPDS.2002.1158262

Return

Native data representation: An efficient wire format for high-performance distributed computing

2002-12-01

PRE

Greg Eisenhauer

Fabián E. Bustamante

Karsten Schwan

DOI：10.1109/TPDS.2002.1158262

Original

Save

Abstract

En 中文

New trends in high-performance software development such as took and component-based approaches have increased the need for flexible and high-performance communication systems. When trying to reap the well-known benefits of these approaches, the question of what communication infrastructure should be used to link the various components arises. In this context, flexibility and high-performance seem to be incompatible goals. Traditional HPC-style communication libraries, such as MPI, offer good performance, but are not intended for loosely-coupled systems. Object- and metadata-based approaches like XML offer the needed plug-and-play flexibility, but with significantly lower performance. We observe that the flexibility and baseline performance of data exchange systems are strongly determined by their wire formats, or by how they represent data for transmission in heterogeneous environments. After examining the performance implications of using a number of different wire formats, we propose an alternative approach for flexible high-performance data exchange, Native Data Representation, and evaluate its current implementation in the Portable Binary I/O library.

Keywords:

high-performance

distributed computing

communication

wire format