arXiv:2503.23895cs.CLcs.AI2025-03被引 28

动态参数检索增强生成,让大模型测试时高效补知识、防幻觉。

Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancement

  • 用轻量翻译模型把文档转成参数知识,动态注入大模型
  • 推理/训练/存储成本大幅降低,且支持即插即用式知识更新
  • 在多个数据集上验证有效,适合需要实时知识增强的场景

检索增强生成(RAG)通过从外部源检索相关文档并融入上下文来提升大语言模型(LLM)的可靠性。然而,随着上下文长度增加,推理成本显著上升,并存在由模型缺乏对应参数知识引发的RAG幻觉问题。一种高效解决方案是在测试时增强LLM的知识。参数化RAG(PRAG)通过将文档嵌入模型参数中实现测试时知识增强,通过离线训练有效降低推理成本。但其高昂的训练与存储开销及有限泛化能力严重限制了实际应用。为此,我们提出动态参数RAG(DyPRAG),利用轻量级参数转换模型高效将文档转化为参数知识。DyPRAG不仅降低推理、训练和存储成本,还能动态生成参数知识,无缝增强LLM知识并以即插即用方式解决知识冲突。在多个数据集上的大量实验表明,DyPRAG具备出色的有效性与泛化能力,提供一种强大且实用的RAG范式,实现更优的知识融合并缓解真实应用场景中的RAG幻觉。代码已开源:https://github.com/Trae1ounG/DyPRAG。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by retrieving relevant documents from external sources and incorporating them into the context. While it improves reliability by providing factual texts, it significantly increases inference costs as context length grows and introduces challenging issue of RAG hallucination, primarily caused by the lack of corresponding parametric knowledge in LLMs. An efficient solution is to enhance the knowledge of LLMs at test-time. Parametric RAG (PRAG) addresses this by embedding document into LLMs parameters to perform test-time knowledge enhancement, effectively reducing inference costs through offline training. However, its high training and storage costs, along with limited generalization ability, significantly restrict its practical adoption. To address these challenges, we propose Dynamic Parametric RAG (DyPRAG), a novel framework that leverages a lightweight parameter translator model to efficiently convert documents into parametric knowledge. DyPRAG not only reduces inference, training, and storage costs but also dynamically generates parametric knowledge, seamlessly enhancing the knowledge of LLMs and resolving knowledge conflicts in a plug-and-play manner at test-time. Extensive experiments on multiple datasets demonstrate the effectiveness and generalization capabilities of DyPRAG, offering a powerful and practical RAG paradigm which enables superior knowledge fusion and mitigates RAG hallucination in real-world applications. Our code is available at https://github.com/Trae1ounG/DyPRAG.

RAG大模型知识增强动态生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。