.png){kind=icon}
返回
A rewrite-based optimizer for Spark
.png){kind=icon}
PRE
AI
DOI：10.1016/j.future.2019.03.044.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Spark is the leading platform for distributed large-scale data processing. Spark's Application Programming Interface (API) has a powerful easy-to-use distributed abstractions similarly related to functional programming (e.g., map, filter, reduce) in several different languages. However, writing an efficient Spark applications is still error-prone, time-consuming, and requires a clear and deep understanding of the inner-workings of Spark. For instance, the same task can be implemented in several different ways, yet the execution time can vary drastically between them. For this, we introduce TaBOS, a rewrite-based optimizer for Spark programs. TaBOS takes a Spark job and automatically generates a state-space of equivalent optimized jobs using a set of semantics-preserving rewrite rules. Then, from the generated state-space, it selects one optimal program based on a predefined strategy. We introduce several selection strategies (e.g., job with maximum number of applied rewrite rules, job with minimum number of heavy operations) for identifying an optimal job from the generated state-space. We evaluate the effectiveness, robustness and speedup gain of our solutions using several case studies. (C) 2019 Elsevier B.V. All rights reserved.
Keyword:
Spark
Optimization
Big data analytics
Source-to-source
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
F
IF:
6.1
论文数:
6.8K
被引数:
2.3W
机构
引用论文
.png){kind=icon}
分享收藏
查看更多内容