让大模型智能体通过交互持续进化，小模型也能超越GPT-4o。
Process Reward Models for LLM Agents: Practical Framework and Directions
- 用轻量级演员-评论家框架，基于蒙特卡洛推演优化策略。
- 30亿参数模型经训练后，在ALFWorld上超越GPT-4o基线。
- 无需明确结果反馈，可从示范中直接学习过程奖励。
我们提出代理过程奖励模型（AgentPRM），一种简单且可扩展的框架，使大语言模型智能体通过交互持续改进。AgentPRM采用轻量级演员-评论家范式，利用蒙特卡洛推演计算奖励目标并优化策略，仅需对现有强化学习人类反馈（RLHF）流程做最小修改，便于大规模集成。此外，我们提出InversePRM，无需显式结果监督即可从示范中直接学习过程奖励。我们还探讨了探索、过程奖励设计与模型预测推理等关键挑战与机遇。在ALFWorld基准上评估显示，使用AgentPRM和InversePRM训练的小型30亿参数模型性能优于强大的GPT-4o基线，并分析了测试时缩放、奖励欺骗等问题。代码已开源：https://github.com/sanjibanc/agent_prm。
原文摘要 · Abstract (English)
We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent_prm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。