.png){kind=icon}
.png){kind=icon}
Return
I/o for LLM inference: a survey of storage and memory bottlenecks
.png){kind=icon}
OA
AI
R
DOI：10.1007/s10462-026-11651-1.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deploying Large Language Models at scale has shifted the dominant bottleneck from compute during training to memory and I/O during inference. As parameter counts reach hundreds of billions and context windows stretch past a million tokens, latency and throughput are limited not by arithmetic but by data movement across the memory hierarchy. This survey decomposes inference I/O into three flows—model weight I/O, Key-Value (KV) cache I/O, and activation I/O—and uses roofline analysis to map each optimization to the memory-hierarchy level it targets. We cover quantization, PagedAttention, FlashAttention, speculative decoding, KV cache compression, and offloading, alongside system-level orchestration (continuous batching, disaggregated prefill-decode, prefix caching) and hardware trends (HBM scaling, CXL, processing-in-memory, unified memory). A composability analysis reveals that stacking optimizations causes the dominant bottleneck to oscillate between weight and KV cache I/O. We close by identifying open problems in unbounded-context scaling, expert caching, edge deployment, and I/O-aware benchmarking.
Keywords:
Large language model inference
Memory bandwidth bottleneck
Roofline analysis
KV cache optimization
Model quantization
Inference serving systems
Journal
IF:
13.9
Papers:
6.1K
Citations:
1.9W
Organization
No organization information available
Cited Papers