.png){kind=icon}
Return
An Efficient DNN Model Serving System using Layer-wise Caching and Direct-Host-Access
.png){kind=icon}
PRE
AI
DOI：10.1145/3774909.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
we can immediately serve the inference requests with low latency. Otherwise, it needs to load the model from host to GPU, adding a significant delay to inference. This article proposes Ignite to minimize cold-start latency while provisioning DL models from host to GPU in server environments. First, we propose LCache to effectively utilize the limited GPU memory for model serving with unique cache replacement policies. We devise layer-wise cache replacement policies tailored for executing DL inferences in a pipelined way. Second, we take advantage of the direct-host-access facility provided by commodity GPUs, allowing access to particular layers of models in the host memory directly from GPU without loading. We show that Ignite can effectively reduce the cold-start latency while increasing the throughput of serving DNN models. When deploying multiple ResNet, BERT, and RoBERTa instances on a DL inference serving system, Ignite shows a significant performance improvement compared to the pipelining technique and stable 99% tail latency.
Keywords:
DNN model serving
layer-wise caching
direct-host-access
Journal
A
IF:
1.8
Papers:
26
Citations:
1.1K
Organization
Cited Papers
.png){kind=icon}
ShareSave
ShareSaveShareSaveShareSave
View more