arXiv:2607.19345cs.CLcs.AI2026-07

通过强化学习提升长文本推理中的关键信息聚焦,减少无效复制。

Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning

  • 设计奖励机制,鼓励模型关注关键证据并避开无关内容。
  • 在长上下文任务中最高提升4.6分,显著减少重复拷贝和推理长度。
  • 适用于需要精准推理的复杂任务,尤其适合长文本场景研究者。

具备逐步推理能力的大语言模型在复杂任务上表现优异,将其扩展至长上下文场景已成为重要前沿。然而我们发现该领域存在关键缺陷:模型在推理过程中大量复制输入文本内容,而非有效解决问题。这种重复复制现象在前沿长上下文LLM中普遍存在,且随上下文长度增加而加剧。通过将提示分离为任务相关的关键证据与无关干扰内容,我们进一步发现根源在于缺乏有效定位——模型盲目复制输入,未能聚焦关键证据的模型更易出错。为此,我们提出GEAR(Grounding Evidence-Aware Reward),一种基于奖励塑形的方法,通过引入与关键证据重叠的奖励和与干扰内容重叠的惩罚,增强准确性信号。为在自然语言数据上应用GEAR,我们构建了自动化证据标注训练数据生成流程。在多个模型规模与基准测试中验证表明,相比标准基于准确率的强化学习,平均性能提升最高达+4.6分,长上下文下增益更大,同时减少重复复制与推理长度。研究揭示,即使长上下文评估转向复杂推理,对关键证据的精准定位仍是核心能力,仍有巨大改进空间。

原文摘要 · Abstract (English)

Large language models that generate step-by-step reasoning traces have achieved strong performance on complex tasks, and extending them to long-context settings has emerged as an important frontier. However, we identify a critical failure mode in this regime: \emph{repetitive copying}, where models extensively copy text from the input into their reasoning traces rather than productively solving the problem. We show that this behavior is pervasive across frontier long-context LLMs and intensifies with context length. By separating each prompt into task-relevant key evidence and irrelevant distractor context, we further show that the root cause is insufficient grounding: models copy from the prompt indiscriminately, and those that fail to focus on key evidence are far more likely to answer incorrectly. Motivated by this diagnosis, we propose GEAR (Grounding Evidence-Aware Reward), a reward shaping method that augments the accuracy signal with a grounding reward for overlap with key evidence and a distractor penalty for overlap with irrelevant context. To enable GEAR on natural-language data, we develop an automated pipeline that constructs evidence-annotated training data from arbitrary documents. We validate GEAR across multiple model scales and benchmarks, showing consistent improvements of up to +4.6 average points over standard RL with accuracy-based rewards, with larger gains at longer contexts, while also reducing repetitive copying and thinking length. Our findings suggest that, even as long-context evaluation shifts from simple retrieval toward complex reasoning, accurate grounding in relevant evidence remains an indispensable capability with substantial room for improvement.

长上下文推理优化强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。