Fine-Grained Dynamic Resource Allocation for Big-Data Applications

Luciano Baresi

doi:10.1109/TSE.2019.2931537

2021-08-01

Luciano Baresi

Alberto Leva

Giovanni Quattrocchi *

DOI：10.1109/TSE.2019.2931537

原文链接

查看原文

摘要

En 中文

Many big-data applications are batch applications that exploit dedicated frameworks to perform massively parallel computations across clusters of machines. The time needed to process the entirety of the inputs represents the application's response time, which can be subject to deadlines. Spark, probably the most famous incarnation of these frameworks today, allocates resources to applications statically at the beginning of the execution and deviations are not managed: to meet the applications' deadlines, resources must be allocated carefully. This paper proposes an extension to Spark, called dynaSpark, that is able to allocate and redistribute resources to applications dynamically to meet deadlines and cope with the execution of unanticipated applications. This work is based on two key enablers: containers, to isolate Spark's parallel executors and allow for the dynamic and fast allocation of resources, and control-theory to govern resource allocation at runtime and obtain required precision and speed. Our evaluation shows that dynaSpark can (i) allocate resources efficiently to execute single applications with respect to set deadlines and (ii) reduce deadline violations (w.r.t. Spark) when executing multiple concurrent applications.

Keyword:

Sparks

Resource management

Dynamic scheduling

Containers

Task analysis

Runtime

Control theory

Distributed architectures

control theory

quality assurance

batch processing systems