.png){kind=icon}
返回
Task Scheduling for Spark Applications With Data Affinity on Heterogeneous Clusters
.png){kind=icon}
PRE
AI
DOI：10.1109/JIOT.2022.3181997.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The Internet of Things (IoT)-enabled applications use sensors and actuators to collect big data, which are processed by big data models, e.g., Spark. Generally, data processing tasks are precedence constrained and the computation results are transmitted to other IoT devices. In this article, we consider the Spark workflow problem of scheduling tasks with data affinity to heterogeneous servers to minimize the maximum completion time. In a Spark instance, jobs are precedence constrained and stages for each job are also precedence constrained. There are a large number of topological stage orders. A balance between task execution times, determined by heterogeneous servers, and transmission times caused by data affinity is difficult to achieve. A scheduling optimization algorithm framework is proposed, which consists of five components: 1) temporal parameter calculation; 2) ready stage adding; 3) task sequencing; 4) resource allocation; and 5) schedule improvement. Strategies for each component are developed. The algorithmic components are statistically calibrated over a comprehensive set of instances. The proposed algorithm is compared to two modified classic algorithms for similar problems on typical scientific workflow instances. The experimental results demonstrate the effectiveness of the proposal for the considered problem.
Keyword:
Task analysis
Sparks
Servers
Scheduling
Schedules
Data communication
Internet of Things
Data affinity
heterogeneous cluster
spark
task scheduling
期刊
IF:
8.9
论文数:
1.4W
被引数:
7.8W
机构
引用论文
SLA-Based Scheduling of Spark Jobs in Hybrid Cloud Computing Environments混合云计算环境下基于SLA的Spark作业调度
PREAI
分享收藏
Synthesis of CeO2/e-HTiNbO5 Nanocomposite and Its Application for Photocatalytic Oxidation Desulfurization.png)
Nano
NanoIF0
PREAI
分享收藏
AWARE-CNN: Automated Workflow for Application-Aware Real-Time Edge Acceleration of CNNsAwate-cnn: 用于CNN的应用程序感知实时边缘加速的自动化工作流
PREAI
分享收藏
查看更多内容