用少量适配器和路由机制高效编码外部知识,降低存储与训练成本。
Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapters
- 用潜空间路由函数组合少量LoRA适配器来编码文档。
- 在四个基准上优于现有PRAG方法,存储开销大幅降低。
- 适合需高效集成外部知识的大型语言模型应用。
参数化检索增强生成(PRAG)通过LoRA适配器将外部知识直接融入模型参数,以降低传统RAG的推理开销。然而,现有方法采用一对一文档编码方案,为每篇文档分配一个专用的LoRA适配器,导致训练与存储成本随文档数量急剧上升,且适配器间因共享知识而高度重叠,效率低下。为此,我们提出Poly-PRAG方法,仅使用一组少量的通用LoRA适配器,并通过潜空间路由函数动态组合这些适配器来编码每篇文档。通过联合训练适配器与路由函数,使每个适配器学习跨文档的共性知识,路由函数则为每篇文档选择最优适配器组合。在四个基准测试上的实验表明,Poly-PRAG在性能上优于多个强基线,同时显著减少存储需求,提供更高效的外部知识注入方式。
原文摘要 · Abstract (English)
Parametric Retrieval-Augmented Generation (PRAG) is a RAG approach that integrates external knowledge directly into model parameters using a LoRA adapter, aiming at reducing the inference cost compared to traditional RAG. However, current PRAG approaches adopt a \textit{one-to-one} document encoding scheme, using a dedicated LoRA adapter for each individual document. This scheme introduces two major limitations: 1) As the number of documents increases, there will be a prohibitive cost for training and storage. 2) The LoRA adapters may largely overlap due to the shared knowledge across documents, making the approach highly inefficient. To overcome these challenges, we propose the Poly-PRAG approach, which uses a small set of LoRA adapters that are able to encode more general knowledge. Each document can be encoded using a combination of them through a latent routing function. By jointly training the LoRA adapters and the latent routing function, each LoRA adapter is able to encode a shared part of the knowledge across documents, and the routing function can select the best combination of adapters for a document. Experimental results on four benchmarks demonstrate the effectiveness of the Poly-PRAG compared to other strong PRAG baselines. In addition, this approach reduces the storage requirement by avoiding the need to store a large number of LoRA adapters and offers a more efficient way to encode external knowledge into LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。