用状态图拓扑传播，为大模型智能体生成无标注密集奖励。
RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Models
- 构建轨迹拓扑状态图，按结构传播奖励
- 四类任务中成功率最高提升29.7%
- 适合需要高效训练的智能体强化学习场景
强化学习在提升大语言模型智能体推理能力方面展现出潜力，但稀疏的终态奖励限制了细粒度优化。过程奖励建模虽可替代，却存在计算开销高、奖励欺骗风险及标注瓶颈问题。我们提出RewardFlow，一种轻量级方法，用于估计智能体推理中的状态级奖励。通过构建捕捉轨迹内在拓扑结构的状态图，实现拓扑感知的奖励传播，从而无须标注即可获得合理且密集的奖励信号。在四种智能体基准测试中，该方法显著优于现有基线：文本任务平均成功率提升6.2%，视觉推理任务在三个模型规模下相较最强基线提升29.7%，DeepResearch任务准确率提高10%，且具备更强鲁棒性与训练效率。代码已开源：https://github.com/tmlr-group/RewardFlow。
原文摘要 · Abstract (English)
Reinforcement learning (RL) shows promise for enhancing LLM agentic reasoning, yet sparse terminal rewards hinder fine-grained optimization. Process reward modeling offers an alternative but incurs high computational costs, reward hacking risks, and annotation bottlenecks. We introduce RewardFlow, a lightweight method for estimating state-level rewards in agentic reasoning. By constructing state graphs that capture the intrinsic topological structure of trajectories, RewardFlow performs topology-aware propagation to estimate each state's contribution to success, yielding principled, annotation-free dense rewards. Used for RL optimization, RewardFlow substantially outperforms prior baselines across four agentic benchmarks: +6.2% average success rate on text-based tasks, +29.7% on visual reasoning over the strongest baseline across three model scales, and +10% accuracy on DeepResearch, with superior robustness and training efficiency. The implementation of RewardFlow is publicly available at https://github.com/tmlr-group/RewardFlow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。