Bi-objective Traffic Optimization in Geo-distributed Data Flows

Anna-Valentini Michailidou

doi:10.1016/j.bdr.2019.04.002

2019-07-01

PRE

Anna-Valentini Michailidou

Anastasios Gounaris *

DOI：10.1016/j.bdr.2019.04.002

原文链接

原文求助

摘要

En 中文

Recently, there have been several proposals in the area of geo-distributed big data processing. In this work, we aim to address a limitation of the existing solutions, namely to optimize task allocation across geographically distributed data centers, in a way that both the total traffic and the running time of the whole processing in complex multi-stage flows are targeted. Apart from proposing concrete efficient solutions for this combinatorial problem, we advocate to take a critical stand on the broadly spread claim that transferring distributed data to a single or fewer places is too costly. In our proposal, we judiciously reduce the participation of some data centers in the flow execution, and we show that, in a wide range of settings, this yields significant benefits. We show that a stochastic solution is superior to a fast greedy one, at the expense of optimization time of up to a few minutes. Compared to a state-of-the-art solution, we manage to decrease total traffic by 44% and running time by 37% on average. In several cases, the improvements can reach 1-2 orders of magnitude. Moreover, we provide evidence that simple heuristics are inferior. Our experimental evaluation comprises both extensive simulations and real runs in Spark. (C) 2019 Elsevier Inc. All rights reserved.

Keyword:

Distributed flow optimization

Latency minimization

Communication minimization

Iridium