MarVeLScaler: A Multi-View Learning-Based Auto-Scaling System for MapReduce

doi:10.1109/TCC.2019.2944916

MarVeLScaler: A Multi-View Learning-Based Auto-Scaling System for MapReduce

2022-01-01

8

PRE

AI

DOI：10.1109/TCC.2019.2944916

Original

Original request for help

Save

Abstract

En 中文

To promote cloud computing from current pay-per-request model to truly pay-per-use, tenants are crying for automatic tools to auto-estimate the amount of resources for MapReduce jobs. Such tools call for accurately quantifying the relationship among workload, resources and completion time. Various prediction models have been proposed. However, none of these models takes virtual machines' (VMs) performance variance during a job's execution into consideration, leading to underestimate the required resources and exceed the job's deadline. To address this problem, we propose a multi-view deep learning model to capture real-time performance variance and automatically scale out the cloud cluster whenever necessary. We implement MarVeLScaler, a prototype system including two useful modules, namely, Scale Estimator and Scale Controller. Scale Estimator preliminarily estimates the required cluster size for a MapReduce job with given a concrete workload and deadline. During the runtime, Scale Controller adjusts the scale of the cluster according to its real-time running status to guarantee the job finished on time. We evaluate the performance of MarVeLScaler based on Hadoop in Alibaba Cloud. Experiments show that MarVeLScaler can provide 98.4 percent accuracy of prediction in determining initial cluster size, and save 30.8 percent of expense while still guaranteeing similar performance compared with the state-of-the-art methods.

Keywords:

Cloud computing

auto-scaling

multi-view neural network

MarVeLScaler: A Multi-View Learning-Based Auto-Scaling System for MapReduce

Abstract

AI Summary

Journal

Organization