.png){kind=icon}
Return
Big data transfer optimization through adaptive parameter tuning
.png){kind=icon}
OA
AI
DOI：10.1016/j.jpdc.2018.05.003.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Obtaining optimal data transfer performance is of utmost importance to today's data-intensive distributed applications and wide-area data replication services. Tuning application-layer protocol parameters such as pipelining, parallelism, and concurrency can significantly increase efficient utilization of the available network bandwidth as well as the end-to-end data transfer performance. However, determining the best settings for these parameters is a challenging problem, as network conditions can vary greatly between sites and over time. Poor protocol tuning can cause either under-or over-utilization of network resources and thus degrade transfer performance. In this paper, we present three novel algorithms for application-layer parameter tuning and transfer scheduling to maximize transfer throughput in wide-area networks. Our algorithms use heuristics to tune the level of control channel pipelining (for small file optimization), the number of parallel data streams per file (for large file optimization), and the number of concurrent file transfers to increase I/O throughput (for all types of files). The proposed algorithms improve the transfer throughput up to 10x compared to the baseline and 7x compared to the state-of-the-art solutions. We also propose adaptive tuning to adjust the values of parameters based on real-time observations. The results show that adaptive tuning can further improve transfer throughput by up to 24% compared to the heuristic approach. (C) 2018 Elsevier Inc. All rights reserved.
Keywords:
Application-level protocol tuning
Throughput optimization
Wide-area data transfers
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
4
Papers:
3.8K
Citations:
4.8K
Organization
Cited Papers
.png){kind=icon}
ShareSave
ShareSaveApplication-Level Optimization of Big Data Transfers through Pipelining, Parallelism and ConcurrencyOAAI
OAAIShareSaveShareSaveShareSaveA Data Throughput Prediction and Optimization Service for Widely Distributed Many-Task ComputingOAAI
OAAIShareSave
View more