.png){kind=icon}
返回
Predictive modelling of MapReduce job performance in cloud environments using machine learning techniques
.png){kind=icon}
OA
AI
DOI：10.1186/s40537-024-00964-z.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Within the Hadoop ecosystem, MapReduce stands as a cornerstone for managing, processing, and mining large-scale datasets. Yet, the absence of efficient solutions for precise estimation of job execution times poses a persistent challenge, impacting task allocation and distribution within Hadoop clusters. In this study, we present a comprehensive machine learning approach for predicting the execution time of MapReduce jobs, encompassing data collection, preprocessing, feature engineering, and model evaluation. Leveraging a rich dataset derived from comprehensive Hadoop MapReduce job traces, we explore the intricate relationship between cluster parameters and job performance. Through a comparative analysis of machine learning models, including linear regression, decision tree, random forest, and gradient-boosted regression trees, we identify the random forest model as the most effective, demonstrating superior predictive accuracy and robustness. Our findings underscore the critical role of features such as data size and resource allocation in determining job performance. With this work, we aim to enhance resource management efficiency and enable more effective utilisation of cloud-based Hadoop clusters for large-scale data processing tasks.
Keyword:
Hadoop
MapReduce
Big data
Performance modelling
Runtime prediction
Machine learning
期刊
IF:
6.4
论文数:
1.5K
被引数:
1.1W
机构
引用论文
Incidence of Dementia in Relation to Genetic Variants at PITX2, ZFHX3, and ApoE ε4 in Atrial Fibrillation Patients房颤患者中与PITX2，ZFHX3和apoeε4遗传变异相关的痴呆发生率
OAAI
OAAI分享收藏Estimating runtime of a job in Hadoop MapReduce在Hadoop MapReduce中估计作业的运行时间.png)
JOURNAL OF BIG DATAOAAI
JOURNAL OF BIG DATAIF6.4
OAAI分享收藏
查看更多内容