.png){kind=icon}
返回
Big data transfer optimization through adaptive parameter tuning
.png){kind=icon}
OA
AI
DOI：10.1016/j.jpdc.2018.05.003.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Obtaining optimal data transfer performance is of utmost importance to today's data-intensive distributed applications and wide-area data replication services. Tuning application-layer protocol parameters such as pipelining, parallelism, and concurrency can significantly increase efficient utilization of the available network bandwidth as well as the end-to-end data transfer performance. However, determining the best settings for these parameters is a challenging problem, as network conditions can vary greatly between sites and over time. Poor protocol tuning can cause either under-or over-utilization of network resources and thus degrade transfer performance. In this paper, we present three novel algorithms for application-layer parameter tuning and transfer scheduling to maximize transfer throughput in wide-area networks. Our algorithms use heuristics to tune the level of control channel pipelining (for small file optimization), the number of parallel data streams per file (for large file optimization), and the number of concurrent file transfers to increase I/O throughput (for all types of files). The proposed algorithms improve the transfer throughput up to 10x compared to the baseline and 7x compared to the state-of-the-art solutions. We also propose adaptive tuning to adjust the values of parameters based on real-time observations. The results show that adaptive tuning can further improve transfer throughput by up to 24% compared to the heuristic approach. (C) 2018 Elsevier Inc. All rights reserved.
Keyword:
Application-level protocol tuning
Throughput optimization
Wide-area data transfers
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4
论文数:
3.8K
被引数:
4.8K
机构
引用论文
.png){kind=icon}
Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and ConcurrencyOAAI
OAAI分享收藏A Data Throughput Prediction and Optimization Service for Widely Distributed Many-Task ComputingOAAI
OAAI分享收藏
查看更多内容