通过动态证据状态奖励，提升长文本推理中证据的精准筛选与利用。
Evidence-State Rewards for Long-Context Reasoning
- 引入可编辑证据记忆，按动作类型设计状态转移奖励
- 在多个数据集上显著降低干扰信息留存率，提升证据充分性
- 适合需要精细推理和证据管理的复杂问答场景
长文本推理要求模型在冗长输入中定位、修正并整合分散的证据。现有长上下文强化学习方法通常仅奖励最终答案或静态证据提取，对中间操作如何改变证据状态缺乏反馈。本文提出Maven框架，采用可编辑的证据记忆机制，定义基于答案的证据状态价值，并对动作级状态转移进行奖励：添加动作根据边际收益和事后贡献计分，链接动作依据证据协同效应奖励，删除动作则因移除误导性证据后答案支持度提升而获益。这些奖励被分配至GRPO中的对应动作区间。在Llama与Qwen模型上，Maven于LongBench v2、LongReason和RULER数据集上均超越仅奖励结果或证据识别的基线，生成更充分的证据集合，且保留的干扰项更少。结果表明，长上下文强化学习应聚焦于有状态的证据导航，而非一次性证据提取。
原文摘要 · Abstract (English)
Long-context reasoning requires models to locate, revise, and synthesize evidence distributed across lengthy inputs. Existing long-context RL methods usually reward final answers or static evidence extraction, offering little feedback on how intermediate actions change the model's evidence state. We propose Maven, a reinforcement learning framework with an editable evidence memory. Maven defines an answer-conditioned evidence-state value and rewards action-level state transitions: add actions are credited by marginal gain and hindsight contribution, link actions by evidence synergy, and drop actions by improved answer support after removing misleading evidence. These rewards are assigned to the corresponding action spans in GRPO. Across Llama and Qwen models on LongBench v2, LongReason, and RULER, Maven outperforms outcome-only RL and evidence-identification baselines, producing more sufficient evidence sets and lower distractor retention. Our results show that long-context RL benefits from optimizing stateful evidence navigation rather than one-shot evidence extraction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。