让边缘设备跑RAG，内存够用且速度更快。
EdgeRAG: Online-Indexed RAG for Edge Devices
- 按聚类动态剪枝+按需生成嵌入，节省内存
- 大尾部聚类预计算嵌入，降低检索延迟
- 适合资源受限的边缘部署场景
在资源受限的边缘设备上部署检索增强生成（RAG）面临内存和算力不足的挑战。本文提出EdgeRAG，通过在聚类内剪枝嵌入，并在检索时按需生成嵌入来缓解内存压力。为避免对大型尾部聚类进行嵌入生成带来的延迟，EdgeRAG预先计算并存储这些聚类的嵌入，同时自适应缓存其余嵌入以减少重复计算，进一步优化延迟。BEIR基准测试结果显示，EdgeRAG相比基线IVF索引显著降低延迟，生成质量相当，同时所有评估数据集均可装入内存。
原文摘要 · Abstract (English)
Deploying Retrieval Augmented Generation (RAG) on resource-constrained edge devices is challenging due to limited memory and processing power. In this work, we propose EdgeRAG which addresses the memory constraint by pruning embeddings within clusters and generating embeddings on-demand during retrieval. To avoid the latency of generating embeddings for large tail clusters, EdgeRAG pre-computes and stores embeddings for these clusters, while adaptively caching remaining embeddings to minimize redundant computations and further optimize latency. The result from BEIR suite shows that EdgeRAG offers significant latency reduction over the baseline IVF index, but with similar generation quality while allowing all of our evaluated datasets to fit into the memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。