.png){kind=icon}
.png){kind=icon}
Return
Distributed Semantic Cache Coordination for Retrieval-Augmented Generation Systems
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-032-07175-0_14.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Retrieval-Augmented Generation (RAG) systems enhance Large Language Models with external knowledge retrieval to produce more accurate responses. In practice, RAG services are often deployed in distributed, multi-tenant environments across geographically separated nodes to serve diverse user communities efficiently. Unfortunately, repeated or semantically similar queries issued by different tenants to different nodes can trigger redundant embedding, retrieval, and LLM inferenceinflating both latency and cost. We introduce a Distributed Semantic Cache (DSC) coordination mechanism to mitigate this inefficiency. Every node keeps a local cache of query embeddings and answers and periodically exchanges a compact Bloom-filter summary of its keys with peers. On a miss, the node consults these summaries and forwards the query embedding only to peers that are likely to hold a semantically similar result, avoiding a central bottleneck and heavy network chatter. We prototype DSC in a discrete-event simulator built with SimPy, FastEmbed, and pybloom_live, modelling a four-node RAG cluster. Compared with a Centralised Exact-match Cache (CEC), DSC delivers up to 580% higher cache-hit rate and makes answers 3.4x faster on average, while adding only 3.6 MB per hour per node of coordination egress traffic per node. These results demonstrate that lightweight, embedding-aware coordination can substantially improve the efficiency and scalability of production-grade RAG deployments.
Keywords:
Semantic Caching
Retrieval-Augmented Generation
Distributed Cache
Blo omFilters
Simulation
Journal
A
IF:
0
Papers:
26
Citations:
0
Organization
No organization information available