.png){kind=icon}
Return
Cola: Cross-Processor Operator Parallelism for Asynchronous Deep Learning Inference
.png){kind=icon}
PRE
AI
DOI：10.1109/TMC.2025.3622698.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Multi-task inference, as a prevalent inference paradigm nowadays, requires deploying multiple deep learning models on the hardware platform to concurrently process inference tasks. Modern platforms are typically equipped with various heterogeneous processors, such as CPU-GPU platform. To reduce resource contention and improve quality of service in the multi-task scenario, existing work has studied cross-processor inference at the fine-grained operator-level. However, it lacks specific optimizations for asynchronous multi-task inference systems. In such systems, tasks arrive dynamically, leading to diverse inference progress for each model. This renders offline optimization strategies based solely on the original computation graph suboptimal or even ineffective. Therefore, we propose a novel framework, Cola, to address the cross-processor operator scheduling for asynchronous tasks. Cola introduces intermediate representation to abstract and simplify such dynamic scheduling problem, considering the impact of task arrival patterns on the inference progress, and employs an efficient two-phase search algorithm. We implemented and validated Cola on a real-world case of intelligent steel structure manufacturing. Cola outperforms the state-of-the-art cross-processor operator scheduling framework in both throughput and resource utilization with highly acceptable runtime overhead.
Keywords:
Edge computing
multi-task inference
cross-processor parallelism
operator scheduling
Journal
IF:
9.2
Papers:
5.9K
Citations:
1.8W
Organization
Cited Papers
ShareSaveCoDL.png)
Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services
Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and ServicesIF0
PREAI
ShareSaveShareSave
E3: Early Exiting with Explainable AI for Real-Time and Accurate DNN Inference in Edge-Cloud Systems
PREAI
ShareSaveShareSaveShareSave
View more