Coinf: QoS-aware DRL-based Inference Task Scheduling Framework with Batching Processing

doi:10.1145/3777373

Coinf: QoS-aware DRL-based Inference Task Scheduling Framework with Batching Processing

2026-01-01

0

PRE

AI

DOI：10.1145/3777373

Original request for help

Save

Abstract

En 中文

The emergence of deploying Deep neural network (DNN) services on edge servers has spurred research into efficiently provisioning inference services. However, previous studies have neglected to consider the implications of different types of DNN and varying quality of service (QoS) requirements on QoS violation rates. In this article, we propose a novel framework, named Coinf, for scheduling heterogeneous DNN inference tasks on edge servers. Coinf has the following four advantages to effectively handle attribute analysis, performance balancing, parallel execution, and model accuracy: (1) It enables efficient profiling of domain-specific attributes of various DNN tasks during the offline stage, achieved by constructing a regression model to predict the end-to-end latency of each task. (2) By utilizing the predicted execution time, Coinf achieves a commendable balance among inference latency, system throughput, and QoS violation rate. (3) It employs emerging deep reinforcement learning (DRL) to aggregate individual DNN tasks into batches, enabling concurrent parallel execution. (4) Coinf preserves the accuracies of the provided DNN models by not modifying them. Numerical experiments are constructed to validate the reliability and efficiency of Coinf in handling heterogeneous inference tasks.

Keywords:

Edge computing

DNN inference

deep reinforcement learning

quality of service

batching policy

Coinf: QoS-aware DRL-based Inference Task Scheduling Framework with Batching Processing

Abstract

Journal

Organization