解决大模型推理中外部知识逐渐失效的问题
Knowledge Integration Decay in Search-Augmented Reasoning of Large Language Models
- 在推理开头和结尾锚定检索知识，防止被前面内容淹没
- 多跳问答和复杂推理任务上性能显著提升
- 无需训练的轻量级方法，适合智能体类大模型使用
现代大语言模型通过搜索增强推理将外部知识融入长链条思维，在复杂任务中表现优异。然而我们发现该范式存在一个关键但未被充分研究的瓶颈——知识融合衰减（KID）：随着推理前文长度增加，模型越来越难以将检索到的信息整合进后续步骤，即使相关信息可用也难发挥作用。为此，我们提出无需训练的推理时策略Self-Anchored Knowledge Encoding（SAKE），通过在推理过程的首尾位置锚定检索知识，防止其被先前上下文覆盖，从而保持其语义完整性。在多跳问答和复杂推理基准上的大量实验表明，SAKE能显著缓解KID现象，提升性能，为智能体类大模型的知识融合提供了一种轻量高效的解决方案。
原文摘要 · Abstract (English)
Modern Large Language Models (LLMs) have demonstrated remarkable capabilities in complex tasks by employing search-augmented reasoning to incorporate external knowledge into long chains of thought. However, we identify a critical yet underexplored bottleneck in this paradigm, termed Knowledge Integration Decay (KID). Specifically, we observe that as the length of reasoning generated before search grows, models increasingly fail to integrate retrieved evidence into subsequent reasoning steps, limiting performance even when relevant information is available. To address this, we propose Self-Anchored Knowledge Encoding (SAKE), a training-free inference-time strategy designed to stabilize knowledge utilization. By anchoring retrieved knowledge at both the beginning and end of the reasoning process, SAKE prevents it from being overshadowed by prior context, thereby preserving its semantic integrity. Extensive experiments on multi-hop QA and complex reasoning benchmarks demonstrate that SAKE significantly mitigates KID and improves performance, offering a lightweight yet effective solution for knowledge integration in agentic LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。