ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

doi:10.1109/TPDS.2024.3431189

ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

2024-10-01

0

PRE

AI

DOI：10.1109/TPDS.2024.3431189

Original

Original request for help

Save

Abstract

En 中文

As deep learning (DL) technologies become ubiquitous, GPU clusters are deployed for inference tasks with consistent service level objectives (SLOs). Efficiently utilizing multiple GPUs is crucial for throughput and cost-effectiveness. This article addresses the challenges posed by dynamic input and NVIDIA MIG in scheduling DL workloads. We present ElasticBatch, a scheduling system that simplifies configuration through bucketization and employs a machine learning-based pipeline to optimize settings. Our experiments demonstrate that ElasticBatch achieves a 50% reduction in GPU instances compared to MIG disablement, increases GPU utilization by 1.4% to 6.5% over an ideal scheduler and significantly reduces profiling time. This research contributes to the discourse on efficient utilization of GPU clusters. ElasticBatch's effectiveness in mitigating challenges posed by dynamic inputs and NVIDIA MIG underscores its potential to optimize GPU cluster performance, providing tangible benefits in terms of reduced instances, increased utilization, and significant time savings in real-world deployment scenarios.

Keywords:

Graphics processing units

Dynamic scheduling

Throughput

Processor scheduling

Pipelines

Costs

Quality of service

MIG

batch inference

scheduling system

machine learning

ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

Abstract

Journal

Organization