.png){kind=icon}
.png){kind=icon}
Return
Network Cost-Aware Geo-Distributed Data Analytics System
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2021.3108893.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Many geo-distributed data analytics (GDA) systems have focused on the network performance-bottleneck: inter-data center network bandwidth to improve performance. Unfortunately, these systems may encounter a cost-bottleneck ($) because they have not considered data transfer cost ($), one of the most expensive and heterogeneous resources in a multi-cloud environment. In this article, we present Kimchi, a network cost-aware GDA system to meet the cost-performance tradeoff by exploiting data transfer cost heterogeneity to avoid the cost-bottleneck. Kimchi determines cost-aware task placement decisions for scheduling tasks given inputs including data transfer cost, network bandwidth, input data size and locations, and desired cost-performance tradeoff preference. In addition, Kimchi is also mindful of data transfer cost in the presence of dynamics. Kimchi has been applied to two common GDA MapReduce models: synchronous barrier and asynchronous push-based shuffle. A Kimchi prototype has been implemented on Spark, and experiments show that it reduces cost by 5% similar to 24% without impacting performance and reduces query execution time by 45% similar to 70% without impacting cost compared to other baseline approaches centralized, vanilla Spark, and bandwidth-aware (e.g., Iridium). More importantly, Kimchi allows applications to explore a much richer cost-performance tradeoff space in a multi-cloud environment.
Keywords:
Task analysis
Data transfer
Bandwidth
Wide area networks
Sparks
Iridium
Distributed databases
Geo-distributed data
multi-DCs
multi cloud providers
data analytics system
Journal
IF:
6
Papers:
5.2K
Citations:
1.1W
