.png){kind=icon}
返回
Cola: Cross-Processor Operator Parallelism for Asynchronous Deep Learning Inference
.png){kind=icon}
PRE
AI
DOI：10.1109/TMC.2025.3622698.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Multi-task inference, as a prevalent inference paradigm nowadays, requires deploying multiple deep learning models on the hardware platform to concurrently process inference tasks. Modern platforms are typically equipped with various heterogeneous processors, such as CPU-GPU platform. To reduce resource contention and improve quality of service in the multi-task scenario, existing work has studied cross-processor inference at the fine-grained operator-level. However, it lacks specific optimizations for asynchronous multi-task inference systems. In such systems, tasks arrive dynamically, leading to diverse inference progress for each model. This renders offline optimization strategies based solely on the original computation graph suboptimal or even ineffective. Therefore, we propose a novel framework, Cola, to address the cross-processor operator scheduling for asynchronous tasks. Cola introduces intermediate representation to abstract and simplify such dynamic scheduling problem, considering the impact of task arrival patterns on the inference progress, and employs an efficient two-phase search algorithm. We implemented and validated Cola on a real-world case of intelligent steel structure manufacturing. Cola outperforms the state-of-the-art cross-processor operator scheduling framework in both throughput and resource utilization with highly acceptable runtime overhead.
Keyword:
Edge computing
multi-task inference
cross-processor parallelism
operator scheduling
期刊
IF:
9.2
论文数:
5.9K
被引数:
1.8W
机构
引用论文
CoDL.png)
Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services
Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and ServicesIF0
PREAI
分享收藏ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile DevicesShuffleNet: 一种用于移动设备的极其高效的卷积神经网络
OAAI
OAAI分享收藏E3: Early Exiting with Explainable AI for Real-Time and Accurate DNN Inference in Edge-Cloud Systems
PREAI
分享收藏分享收藏分享收藏
查看更多内容