通过删除每步搜索并评估影响，实现对多轮推理过程的精准奖励分配。
LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
- 基于反向留一法，量化每一步搜索对最终答案的影响。
- 在7个数据集上平均准确率0.326，比最强基线高0.053。
- 无需额外模型或人工标注，适合强化学习驱动的多轮搜索系统。
多轮搜索推理中的强化学习通常依赖最终结果奖励，无法区分中间步骤的有效性。本文提出LOTAPO，一种基于反向留一法的自生成过程监督方法。对每一轮搜索，将该轮及其检索结果替换为[DELETE]占位符，测量策略对正确答案的平均对数似然变化，以此估计该轮贡献，同时保留后续所有交互。该方法进一步引入符号一致性门控机制，仅保留方向与原始归因分数一致的归因值。无需额外奖励模型、教师、验证器或大模型评判者。在7个本地检索的知识密集型问答数据集上，平均精确匹配率达0.326，优于最强基线IGPO的0.273。消融实验表明，反向归因与符号一致性门控具有互补优势，证明策略衍生的回溯归因可有效提供多轮搜索代理的过程监督。
原文摘要 · Abstract (English)
Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LOTAPO , a self-generated process-supervision method based on backward leave-one-turn attribution. For each search turn, LOTAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder and measures the resulting change in the current policy's mean log-likelihood of the gold answer. This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions, allowing early evidence to be evaluated in the complete reasoning context. LOTAPO further applies sign-consistency gating, retaining only normalized process advantages whose directions agree with their raw attribution scores. The method requires no additional reward model, teacher, verifier, or LLM-as-a-Judge. Across seven knowledge-intensive question-answering datasets with local retrieval, LOTAPO achieves an average exact-match score of 0.326, outperforming the strongest step-reward baseline, IGPO, by 0.053. Ablations show complementary benefits from backward attribution and sign-consistency gating, demonstrating that policy-derived retrospective attribution can provide effective process supervision for multi-turn search agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。