.png){kind=icon}
.png){kind=icon}
Return
Workload-Adapted Resource Allocation for LLM Distributed Serving in Serverless Clusters
.png){kind=icon}
PRE
AI
DOI：10.1109/TPDS.2026.3683122.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Large language models increasingly rely on pipeline parallelism for distributed inference, but existing systems face critical challenges in serverless environments: heterogeneous request distributions across pipeline stages and unpredictable workload patterns requiring rapid elasticity. Traditional static resource allocation fails to address pipeline-specific bottlenecks and cold start delays inherent in serverless architectures. We propose <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Quart</small>, a workload-adapted resource allocation system for LLM distributed serving in serverless clusters. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Quart</small> introduces pipeline-aware resource management through: (1) latency-aware critical stage identification using coefficient of variation (CV)-based burst propagation analysis, (2) dynamic replica allocation with proportional-integral-derivative (PID) control for congested stages, and (3) hierarchical parameter caching with copy-on-write mechanisms enabling sub-second serverless scaling. The system addresses serverless-specific challenges through cache-aware scheduling that maintains model parameters in memory, eliminating disk I/O overhead during rapid scaling events. Evaluation with real-world workloads shows <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Quart</small> reduces average response latency by up to 87.1% compared to existing serverless inference systems while achieving 2.37x improvement in goodput.
Keywords:
Pipeline inference
large language model
serverless
Journal
IF:
6
Papers:
5.2K
Citations:
1.1W
View more organizations