发现大模型会忽视上下文后半部分信息，提出新评估框架揭示此偏差。
"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models
- 构建新评估框架CoPE，量化模型对上下文与参数知识的利用程度。
- 发现模型存在'丢失在后期'现象，后半段信息常被忽略，影响理解准确率。
- 提示词优化可缓解该问题，尤其在摘要任务中减少幻觉、提升事实一致性。
大型语言模型虽能调用上下文和参数化知识，但其优先级与整合机制尚不明确。本文提出CoPE评估框架，系统测量不同模型与语言下上下文知识（CK）与参数知识（PK）的表现。基于英、西、丹三语的MultiWikiAtomic数据集，分析模型在开放问答中如何整合上下文、优先处理信息并融合参数知识。研究发现模型存在‘丢失在后期’现象：后半段上下文信息常被忽视，表现出显著位置偏差，影响上下文对齐。进一步发现，推理模型及使用思维链（CoT）提示的非推理模型反而更少利用上下文，且无法缓解该偏差；特别地，CoT导致召回率下降、响应变短，加剧上下文对齐缺失。基于此，设计提示优化方法有效利用输入上下文。案例研究显示，在摘要任务中引入上下文知识指导的提示，可提升事实准确性并减少幻觉。
原文摘要 · Abstract (English)
Large language models are capable of leveraging both contextual and parametric knowledge but how they prioritize and integrate these sources remains underexplored. We introduce CoPE, a novel evaluation framework that systematically measures contextual knowledge (CK) and parametric knowledge (PK) across models and languages. Using our MultiWikiAtomic dataset in English, Spanish, and Danish, we analyze how large language models (LLMs) integrate context, prioritize information, and incorporate PK in open-ended question answering. Our analysis uncovers a phenomenon we call lost-in-the-later, where LLMs tend to overlook or deprioritize information that appears later in a given context, revealing a strong positional bias that affects contextual grounding. We further find that reasoning models, as well as non-reasoning models prompted with chain-of-thought (CoT), use context even less than non-reasoning models without CoT and fail to mitigate the lost-in-the-later effect. CoT prompting, in particular, results in lower recall and shorter responses, leading to degraded contextual grounding. Based on these insights, we design prompt-based methods to effectively leverage input context. A case study applying CoPE to summarization demonstrates that CK-informed prompting improves factual grounding and reduces hallucination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。