Towards Efficient Graph Processing in Geo-Distributed Data Centers

doi:10.1109/TPDS.2024.3453872

2024-11-01

PRE

DOI：10.1109/TPDS.2024.3453872

原文链接

原文求助

摘要

En 中文

Iterative graph processing is widely used as a significant paradigm for large-scale data analysis. In many global businesses of multinational enterprises, graph-structure data is usually geographically distributed in different regions to support low-latency services. Geo-distributed graph processing suffers from the Wide Area Networks (WANs) with scarce and heterogeneous bandwidth, thus essentially differs from traditional distributed graph processing. In this paper, we propose RAGraph, a Region-Aware framework for geo-distributed graph processing. At the core of RAGraph, we design a region-aware graph processing framework that allows advancing inefficient global updates locally and enables sensible coordination-free message interactions and flexible replaceable communication module. In terms of graph data preprocessing, RAGraph introduces a contribution-driven edge migration algorithm to effectively utilize network resources. RAGraph also contains an adaptive hierarchical message interaction engine to switch interaction modes adaptively based on network heterogeneity and fluctuation, and a discrepancy-aware message filtering strategy to filter important messages. Experimental results show that RAGraph can achieve an average speedup of 9.7x (up to 98x) and an average WAN cost reduction of 78.5$% (up to 97.3%) compared with state-of-the-art systems.

Keyword:

Data centers

Wide area networks

Bandwidth

Computational modeling

Iterative algorithms

Fluctuations

Measurement

Graph processing

geo-distributed data centers

heterogeneous network