.png){kind=icon}
返回
RAGCache Efficient Knowledge Caching for Retrieval-Augmented Generation
.png){kind=icon}
PRE
AI
DOI：10.1145/3768628.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Retrieval-Augmented Generation (RAG) has demonstrated substantial advancements in various natural language processing tasks by integrating the strengths of large language models (LLMs) and external knowledge databases. However, the retrieval step introduces long sequence generation and extra data dependency, resulting in long end-to-end latency. Our analysis benchmarks current RAG systems and reveals that, while the retrieval step poses performance challenges, it also offers optimization opportunities through its retrieval pattern and streaming search behavior. We propose RAGCache, a latency-optimized serving system tailored for RAG. RAGCache leverages the retrieval pattern to organize and cache the intermediate states of retrieved knowledge in a knowledge tree across the GPU and host memory hierarchy, reducing LLM generation time. RAGCache employs dynamic speculative pipelining to exploit the streaming search behavior, overlapping retrieval with LLM generation to minimize end-to-end latency. We implement RAGCache based on vLLM and Faiss, and evaluate it on both open-source and production datasets. Experimental results demonstrate that RAGCache reduces the time to first token (TTFT) by up to 4 & times; and improves the throughput by up to 2.1 & times; compared to vLLM integrated with Faiss.
Keyword:
Retrieval-augmented generation
LLM inference
caching
期刊
A
IF:
1.8
论文数:
26
被引数:
1.1K
机构
引用论文
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsGQA: 从多头检查点训练广义多查询转换器模型
OAAI
.png){kind=icon}
OAAI分享收藏Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step QuestionsOAAI
OAAI分享收藏Efficient Memory Management for Large Language Model Serving with PagedAttention使用PagedAttention服务的大型语言模型的有效内存管理OAAI
OAAI分享收藏
查看更多内容