An Efficient DNN Model Serving System using Layer-wise Caching and Direct-Host-Access

doi:10.1145/3774909

2026-02-01

PRE

Jinwoo Jeong

Jeongseob Ahn *

DOI：10.1145/3774909

原文链接

原文求助

摘要

En 中文

我们可以立即以低延迟服务推理请求。否则，它需要从主机加载模型到GPU，为推理增加显著的延迟。本文提出Ignite以最小化冷启动延迟，同时为服务器环境中的DL模型提供主机到GPU的配置。首先，我们提出LCache，通过独特的缓存替换策略有效利用有限的GPU内存进行模型服务。我们设计了分层缓存替换策略，以适应以流水线方式执行DL推理。其次，我们利用消费级GPU提供的直接主机访问设施，允许GPU直接从主机内存访问模型的特定层而无需加载。我们表明，Ignite可以有效减少冷启动延迟，同时提高DNN模型服务的吞吐量。当在DL推理服务系统中部署多个ResNet、BERT和RoBERTa实例时，Ignite相比流水线技术显示出显著性能提升和稳定的99%尾延迟。

Keyword:

DNN model serving

layer-wise caching

direct-host-access

期刊

ACM Transactions on Computer Systems

IF:

1.8

论文数:

被引数:

1.1K

机构

korea university

学者数:

4.9K

论文数: 2.2K

被引数: 1

引用论文

Large graph convolutional network training with GPU-oriented data communication architecture

Proceedings of the VLDB Endowment

IF0

2021-10-27

OAAI

Seung Won Min; Kun Wu; Sitao Huang; Mert Hidayetoğlu; Jinjun Xiong; Eiman Ebrahimi; Deming Chen; Wen-mei Hwu

DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference

2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA)

IF0

2020-05-01

OAAI

Udit Gupta; Samuel Hsia; Vikram Saraph; Xiaodong Wang; Brandon Reagen; Gu-Yeon Wei; Hsien-Hsin S. Lee; David Brooks; Carole-Jean Wu

TwinPilots: A New Computing Paradigm for GPU-CPU Parallel LLM Inference

Proceedings of the 17th ACM International Systems and Storage Conference on ZZZ

IF0

2024-09-16

PREAI

Yu,Chengye; Wang,Tianyu; Shao,Zili; Zhu,Linjie; Zhou,Xu; Jiang,Song

Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access

Proceedings of the Eighteenth European Conference on Computer Systems

IF0

2023-05-08

PREAI

Jinwoo Jeong; Seungsu Baek; Jeongseob Ahn

IF0

PREAI

A Configurable Cloud-Scale DNN Processor for Real-Time AI

2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)

IF0

2018-06-01

PREAI

Jeremy Fowers; Kalin Ovtcharov; Michael Papamichael; Todd Massengill; Ming Liu; Daniel Lo; Shlomi Alkalay; Michael Haselman; Logan Adams; Mahdi Ghandi; Stephen Heil; Prerak Patel; Adam Sapek; Gabriel Weisz; Lisa Woods; Sitaram Lanka; Steven K. Reinhardt; Adrian M. Caulfield; Eric S. Chung; Doug Burger

Transformers: State-of-the-Art Natural Language Processing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

IF0

2020-01-01

OAAI

Thomas Wolf; Lysandre Debut; Victor Sanh; Julien Chaumond; Clement Delangue; Anthony Moi; Pierric Cistac; Tim Rault; Remi Louf; Morgan Funtowicz; Joe Davison; Sam Shleifer; Patrick von Platen; Clara Ma; Yacine Jernite; Julien Plu; Canwen Xu; Teven Le Scao; Sylvain Gugger; Mariama Drame; Quentin Lhoest; Alexander Rush

Proceedings of the ACMSE 2018 Conference

IF0

2018-03-29

PREAI

IF0

PREAI

GSLICE

Proceedings of the 11th ACM Symposium on Cloud Computing

IF0

2020-10-12

OAAI

Aditya Dhakal; Sameer G Kulkarni; K. K. Ramakrishnan

查看更多内容