.png){kind=icon}
返回
DistMind: Efficient Resource Disaggregation for Deep Learning Workloads
.png){kind=icon}
PRE
AI
DOI：10.1109/TNET.2024.3355010.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Deep learning (DL) systems suffer from low resource utilization due to 1) monolithic server model that tightly couples compute and memory; and 2) limited sharing between different inference applications, and across inference and training, because of strict service level objectives (SLOs). To address this problem, we present DistMind, a disaggregated DL system that enables efficient multiplexing of DL applications with near-optimal resource utilization. DistMind decouples compute from host memory, and exposes the abstractions of a GPU pool and a memory pool, each of which can be independently provisioned. The key challenge is to dynamically allocate GPU resources to different applications based on their real-time demands while meeting strict SLOs. We tackle this challenge by exploiting the power of high-speed 100 Gbps networks, and design three-stage pipelining, cache-aware load balancing, and DNN-aware sharding mechanisms based on the characteristics of DL workloads, to achieve millisecond-scale application loading overhead and improve system efficiency. We have implemented a prototype of DistMind and integrated it with PyTorch. Experimental results on AWS EC2 show that DistMind achieves near 100% resource utilization, and compared with NVIDIA MPS and Ray, DistMind improves the throughput by up to 279% and reduces the inference latency by up to 94%.
Keyword:
Machine learning systems
resource disaggregation
resource management
scheduling
deep learning
期刊
I
IF:
3.6
论文数:
4.4K
被引数:
9.5K
机构
引用论文
Field sampling bias, museum collections and completeness of the fossil record野外采样偏差，博物馆收藏和化石记录的完整性.png)
Lethaia
OAAI
LethaiaIF0
OAAI分享收藏Effects of Perinatal Progesterone on Mouse Mammary Differentiation In Vitro:
Brief Communication
2
3
PREAI
分享收藏Microstructure and dielectric properties of CaCu3Ti4O12 ceramics by lanthanum/europium co-dopingMRS Advances
MRS AdvancesIF0
PREAI
分享收藏X-ray crystallographic and solution studies of the pentamethyldiethylenetriamine and tetramethylethylenediamine adducts of lithium diphenylphosphidePolyhedron
PolyhedronIF0
PREAI
分享收藏分享收藏
查看更多内容