.png){kind=icon}
返回
A novel framework for generic Spark workload characterization and similar pattern recognition using machine learning
.png){kind=icon}
OA
AI
DOI：10.1016/j.jpdc.2024.104881.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Comprehensive workload characterization plays a pivotal role in comprehending Spark applications, as it enables the analysis of diverse aspects and behaviors. This understanding is indispensable for devising downstream tuning objectives, such as performance improvement. To address this pivotal issue, our work introduces a novel and scalable framework for generic Spark workload characterization, complemented by consistent geometric measurements. The presented approach aims to build robust workload descriptors by profiling only quantitative metrics at the application task -level, in a non -intrusive manner. We expand our framework for downstream workload pattern recognition by incorporating unsupervised machine learning techniques: clustering algorithms and feature selection. These techniques significantly improve the process of grouping similar workloads without relying on predefined labels. We effectively recognize 24 representative Spark workloads from diverse domains, including SQL, machine learning, web search, graph, and micro -benchmarks, available in HiBench. Our framework achieves a high accuracy F -Measure score of up to 90.9% and a Normalized Mutual Information of up to 94.5% in similar workload pattern recognition. These scores significantly outperform the results obtained in a comparative analysis with an established workload characterization approach in the literature.
Keyword:
Big data
Workload characterization
Apache spark
Pattern recognition
Machine learning
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4
论文数:
3.8K
被引数:
4.8K
机构
引用论文
A comprehensive performance analysis of Apache Hadoop and Apache Spark for large scale data sets using HiBench.png)
JOURNAL OF BIG DATAOAAI
JOURNAL OF BIG DATAIF6.4
OAAI分享收藏
查看更多内容