TOPSERVE: Task-Operator Co-scheduling for Efficient Multi-DNN Inference Serving on GPUs

Ao Chen

doi:10.1007/978-3-031-99857-7_21

TOPSERVE: Task-Operator Co-scheduling for Efficient Multi-DNN Inference Serving on GPUs

2026-01-01

0

PRE

AI

DOI：10.1007/978-3-031-99857-7_21

Original

Original request for help

Save

Abstract

En 中文

Emerging intelligent applications often require collaborative inference from multiple deep neural networks (multi-DNNs) to support complex tasks like augmented and virtual reality. However, efficiently serving multi-DNNs is challenging due to heterogeneous model structures, parallelism strategies, and dynamic batching behaviors. Existing methods either use online task-level scheduling for batched inference or offline operator-level scheduling to optimize concurrency. These approaches, limited to a single perspective, may lead to sub-optimal performance in evolving multi-DNN serving scenarios. In this paper, we present TopServe, an efficient multi-DNN serving system that integrates dynamic batching with adaptive inter-operator parallelization strategies. During the offline phase, TopServe partitions the multi-DNN model into balanced subgraphs and generates candidate operator scheduling strategies. During the online phase, TopServe performs task-operator co-scheduling, combining effective batching with optimized operator parallelization. Our extensive evaluation shows that TopServe can significantly reduce the average latency and improve the throughput compared to state-of-the-art solutions.

Keywords:

Deep Learning Serving Systems

Multi-DNN Inference Serving

Task-Operator Co-Scheduling

TOPSERVE: Task-Operator Co-scheduling for Efficient Multi-DNN Inference Serving on GPUs

Abstract

Journal

Organization