Splitwise: Efficient Generative LLM Inference Using Phase Splitting

doi:10.1109/MM.2025.3575361

Splitwise: Efficient Generative LLM Inference Using Phase Splitting

0

PRE

AI

DOI：10.1109/MM.2025.3575361

Original

Original request for help

Save

Abstract

En 中文

Generative large language model (LLM) applications are rapidly growing, leading to widespread deployment of expensive, power-hungry GPUs. Growing power demands of artificial intelligence (AI) in the cloud industry has become a global problem.5 Our analysis shows that LLM inference involves two distinct phases: a compute-intensive prefill phase and a memory-intensive decode phase, each with different resource needs. Running them together introduces inefficient scheduling. Furthermore, unlike the prefill phase, the decode phase can run on lower-cost and lower-power hardware. Building on these insights, we propose Splitwise, a scheduling technique that splits prefill and decode phases across different machines to achieve better throughput. Additionally, Splitwise allows phase-specific hardware optimization. By efficiently transferring request state between machines, Splitwise achieves up to 2.35× more throughput within the same power and cost budgets, or 1.4× higher throughput at 20% lower cost and same power.

Keywords:

Hardware

Throughput

Costs

Artificial intelligence

Oral communication

Encoding

Bandwidth

Production

Processor scheduling

Power demand

Large language models

Generative AI

Splitwise: Efficient Generative LLM Inference Using Phase Splitting

Abstract

Journal

Organization