arXiv:2607.07435cs.LGcs.AI2026-07

让智能体在真实世界中高效执行任务,既要避免错误路径,又要快速达成目标。

RLVP: Penalize the Path, Reward the Outcome

论文配图:RLVP: Penalize the Path, Reward the Outcome
图 1 · 摘自论文原文
  • 通过惩罚错误路径、奖励最终结果来提升学习效率
  • 在少样本情况下实现接近零违规的成功率
  • 适合需要高可靠性的真实场景任务部署

在真实世界中(如拨打电话),代理需从昂贵且不可逆的交互中在线学习,而非依赖廉价模拟。首先,可部署性不仅取决于结果,还取决于行为路径:必须遵守不重复拨打无人应答用户、遵守营业时间等结果无法体现的约束。其次,由于每次交互成本高,必须从极少样本中高效学习。基于可验证奖励的强化学习(RLVR)仅关注结果,忽略路径,且在全失败组中相对优势归零,浪费资源。试图通过奖励进展来丰富监督,但进展难以验证。相反,真实环境可低成本检测错误动作。由于组内相对优势等价于组内方差,密集信号仅在结果缺乏方差时才有用。可验证的路径惩罚能可靠提供所需方差,而进展奖励仅在部分进展可达时有效。因此,“惩罚路径、奖励结果”的策略在极少违规下实现高成功率,而仅奖励结果的方法几乎每轮都会违反约束。本文提出四项有效惩罚的设计规则,包括避免因单独使用惩罚导致的“静止陷阱”问题。

原文摘要 · Abstract (English)

Agents acting on our behalf in the real world (e.g. placing phone calls) must learn online from costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on the path, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples. Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiable penalty on the path meets this condition reliably, while a progress potential helps only where partial progress is reachable. The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of the inaction trap that arises when a penalty is used in isolation.

强化学习智能体路径约束高效学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。