.png){kind=icon}
返回
BatOpt: Optimizing GPU-Based Deep Learning Inference Using Dynamic Batch Processing
.png){kind=icon}
PRE
AI
DOI：10.1109/TCC.2024.3350561.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep learning (DL) has been applied in billions of mobile devices due to its astonishing performance in image, text, and audio processing. However, limited by the computing capability of mobile devices, a large amount of DL inference tasks need to be offloaded to edge or cloud servers, which makes powerful GPU servers are struggling to ensure the quality of service(QoS). To better utilize the highly parallel computing architecture of GPU to improve the QoS, we propose BatOpt, a framework that uses dynamic batch processing to strike a good balance between service latency and GPU memory usage in DL inference services. Specifically, BatOpt innovatively models the DL inference service as a M/G(a,b)/1/N queue, with the consideration of stochastic task arrivals, which enables it to predict the service latency accurately in different system states. Furthermore, we propose an optimization algorithm to trade off the service latency and GPU memory usage in different system states by analyzing the queueing model. We have implemented BatOpt on Pytorch and evaluated it on an RTX 2080 GPU using real DL models. BatOpt brings up to 31x and 4.3x times performance boost in terms of service latency, compared to single-input and fixed-batch-size strategies, respectively. And BatOpt's maximum GPU memory usage is only 0.3x that of greedy-dynamic-batch-size strategy on the premise of the same service latency.
Keyword:
Graphics processing units
Task analysis
Servers
Throughput
Computational modeling
Mobile handsets
Stochastic processes
Deep learning
GPU servers
batch processing
stochastic process
期刊
I
IF:
5
论文数:
1.8K
被引数:
4.3K
机构
引用论文
Regulation and Functions of the lms Homeobox Gene during Development of Embryonic Lateral Transverse Muscles and Direct Flight Muscles in Drosophila.png)
PLoS ONE
OAAI
PLoS ONEIF0
OAAI分享收藏A Survey on End-Edge-Cloud Orchestrated Network Computing Paradigms: Transparent Computing, Mobile Edge Computing, Fog Computing, and Cloudlet端边云编排网络计算范例调查: 透明计算、移动边缘计算、雾计算和Cloudlet
PREAI
分享收藏分享收藏分享收藏Damage Detection and Level Classification of Roof Damage After Typhoon Faxai Based on Aerial Photo and Deep Learning基于航拍照片和深度学习的Faxai台风后顶板损伤检测与等级分类
OAAI
IF0
OAAI分享收藏
查看更多内容