无需验证器,让大模型智能体自动评估每一步的贡献。
Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifiers

- 用最终答案聚类构建潜在目标状态,实现中间步骤的精准奖励分配。
- 在7个搜索增强型任务上超越无验证器基线,接近有标注监督的效果。
- 适合缺乏人工标注或专用验证器的长时序智能体训练场景。
长周期大模型智能体依赖中间信息收集步骤,但训练反馈通常仅在最终答案处获得,因过程级奖励需高质量人工标注。现有逐轮奖励方法依赖答案监督或稳定的任务特定验证器;而无标签强化学习虽可提取输出分布中的自信号,但多作用于答案或轨迹层面,无法对中间步骤进行信用分配。我们提出自诱导结果势(SIOP),将最终答案的语义聚类视为潜在未来结果状态,用于基于势能的逐轮信用分配。针对每个查询,SIOP生成多个回溯路径,将最终答案聚类为语义结果模式,并构建可靠性感知的目标分布。通过可计算的聚类级近似,奖励使可靠未来状态后验支持增加的步骤。该方法将信息势能塑造从黄金答案监督推广至无任务特定验证器的场景,同时避免标准GRPO使用的广播式回溯级优势。我们形式化该框架,刻画其在有监督黄金答案下的极限表现,并证明SIOP在七个搜索增强型代理推理基准上平均性能优于无验证器的基线,接近黄金监督基线。代码已开源:https://github.com/dl-m9/SIOP.git。
原文摘要 · Abstract (English)
Long-horizon LLM agents depend on intermediate information-gathering turns, yet training feedback is usually observed only at the final answer, because process-level rewards require high-quality human annotation. Existing turn-level shaping methods reward turns that increase the likelihood of a gold answer, but they require answer supervision or stable task-specific verifiers. Conversely, label-free RL methods extract self-signals from output distributions, but mainly at the answer or trajectory level and therefore cannot assign credit to intermediate turns. We propose Self-Induced Outcome Potential (SIOP), which treats semantic clusters of final answers as latent future outcome states for potential-based turn-level credit assignment. For each query, SIOP samples multiple rollouts, clusters final answers into semantic outcome modes, and builds a reliability-aware target distribution over these states. It then rewards turns for increasing posterior support for reliable future states using a tractable cluster-level approximation. The objective generalizes information-potential shaping from gold-answer supervision to settings without task-specific gold verifiers while avoiding the broadcasted rollout-level advantages used by standard GRPO. We formalize the framework, characterize its supervised gold-answer limit, and show that SIOP improves average performance over verifier-free outcome-level baselines on seven search-augmented agentic reasoning benchmarks while approaching a gold-supervised outcome baseline. Code is available at https://github.com/dl-m9/SIOP.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。