对比参数化与传统检索增强生成,发现混合使用效果最佳。
Understanding Parametric Knowledge Injection in Retrieval-Augmented Generation
- 用可训练参数代替文本拼接注入知识,降低长文本负担。
- 混合方法PT-RAG在准确率上优于纯参数或纯文本方式。
- 适合需要高鲁棒性与抗干扰能力的问答系统场景。
上下文感知生成支撑了大模型的诸多应用,包括长文档问答、对话个性化和检索增强生成(RAG)。然而,传统的基于标记的上下文拼接在处理长输入时成本高昂,且在极长上下文下信息易丢失。近期研究探索了上下文参数化方法,将上下文编码为轻量级可训练参数(如LoRA适配器),注入冻结的大语言模型中。将此思想扩展至检索到的证据,形成参数化RAG(P-RAG),通过参数更新而非标记级注意力来融合知识。本文系统研究这一新兴范式——参数化知识注入。首先,在答案存在准确性下重新评估P-RAG,发现其并未始终优于标准的基于标记的RAG(T-RAG),而结合两者(PT-RAG)取得最佳整体性能。其次,构建了一个包含最新知识的QA基准,超越模型内部记忆,实现可控分析。表征与机制结果表明,参数表示捕捉文档级语义,主要影响深层前馈计算,提供高层指导但证据整合有限。最后,在关键RAG挑战下评估参数注入,显示其在知识冲突下更可信,对检索噪声更具鲁棒性,并在问答之外任务中表现良好。研究澄清了参数化RAG的优势与局限,为未来检索增强大模型系统提供实用指导。
原文摘要 · Abstract (English)
Context-grounded generation underpins many LLM applications, including long-document question answering (QA), conversational personalization, and retrieval-augmented generation (RAG). However, classic token-based context concatenation is costly for long inputs and can be lost in the middle at extreme context lengths. Recent work explores context parameterization, which encodes context into lightweight trainable parameters (e.g., LoRA adapters) injected into a frozen LLM. Extending this idea to retrieved evidence yields parametric RAG (P-RAG), which incorporates knowledge via parameter updates rather than token-level attention. In this paper, we present a systematic study of this emerging RAG paradigm-parametric knowledge injection. First, we reassess P-RAG under answer-presence accuracy and show that it does not consistently outperform standard token-based RAG (T-RAG), while combining both (PT-RAG) achieves the best overall performance. Second, we introduce a QA benchmark with up-to-date knowledge beyond the LLM's internal memory to enable controlled analysis. Our representational and mechanistic results indicate that parametric representations capture document-level semantics and primarily influence deeper feed-forward computations, providing high-level guidance but limited evidence consolidation. Finally, we evaluate parametric injection under key RAG challenges, demonstrating improved faithfulness under knowledge conflicts, stronger robustness to retrieval noise, and solid generalization to tasks beyond QA. Our findings clarify the strengths and limitations of parametric RAG and provide practical guidance for future retrieval-augmented LLM systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。