研究大模型如何依赖外部信息，发现其严重偏向外存而忽视自身知识。
Quantifying reliance on external information over parametric knowledge during Retrieval Augmented Generation (RAG) using mechanistic analysis
- 用因果中介分析证明模型提问时极少使用自身参数化知识
- 注意力分析显示模型仅从检索内容获取关键信息，而非问题本身
- 该现象在大模型和小模型中均存在，揭示RAG的普遍性捷径行为
检索增强生成（RAG）在问答与信息检索等自然语言任务中广泛应用，但语言模型如何利用非参数化外部记忆仍不明确。本文通过机制分析发现，语言模型存在明显‘捷径’效应，强烈依赖检索到的上下文回答问题，几乎不使用自身先验知识。我们提出：(a) 因果中介分析，证明模型在回答问题时极少利用参数化记忆；(b) 注意力贡献与击穿实验，表明最后一个标记的残差流仅从RAG上下文中的词元获得信息，而非问题中的主题词元。该显著‘捷径’行为在大模型（如LlaMa）和小模型（如Phi）中均成立。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) is a widely used approach for leveraging external context in several natural language applications such as question answering and information retrieval. Yet, the exact nature in which a Language Model (LM) leverages this non-parametric memory or retrieved context isn't clearly understood. This paper mechanistically examines the RAG pipeline to highlight that LMs demonstrate a "shortcut'' effect and have a strong bias towards utilizing the retrieved context to answer questions, while relying minimally on model priors. We propose (a) Causal Mediation Analysis; for proving that parametric memory is minimally utilized when answering a question and (b) Attention Contributions and Knockouts for showing the last token residual stream do not get enriched from the subject token in the question, but gets enriched from tokens of RAG-context. We find this pronounced "shortcut'' behaviour to be true across both LLMs (e.g.,LlaMa) and SLMs (e.g., Phi)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。