.png){kind=icon}
返回
Efficient Performance Prediction for Apache Spark
.png){kind=icon}
PRE
AI
DOI：10.1016/j.jpdc.2020.10.010.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Spark is a more efficient distributed big data processing framework following Hadoop. It provides users with more than 180 adjustable configuration parameters, and how to choose the optimal configuration automatically to make the Spark application run effectively is challenging. The key to address the above challenge is having the ability to predict the performance of Spark applications in different configurations. This paper proposes a new approach based on Adaboost, which can efficiently and accurately predict the performance of a given application with a given Spark configuration. In our approach, Adaboost is used to build a set of performance models at the stage-level for Spark. To minimize the overhead of the modeling, we use the classic projective sampling, a data mining technique that allows us to collect as few training samples as possible while meeting the accuracy requirements. We evaluate the proposed approach on six typical Spark benchmarks with five input datasets. The experimental results show that our approach is less than the previously proposed approach in prediction error and cost. (C) 2020 Elsevier Inc. All rights reserved.
Keyword:
Performance prediction
Spark
System configuration
Adaboost
Projective sampling
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4
论文数:
3.8K
被引数:
4.8K
机构
引用论文
Discussion: The measurement of particle elongation and flakiness: a critical discussion of British Standard and other methods讨论: 颗粒伸长率和片状度的测量: 对英国标准和其他方法的批判性讨论
.png){kind=icon}
PREAI
分享收藏
Capacity optimization of hybrid renewable energy system considering part-load ratio and resource endowment考虑部分负荷率和资源禀赋的混合可再生能源系统容量优化
PREAI
分享收藏分享收藏 没有更多内容
没有更多内容