arXiv:2608.17423cs.ROcs.LG2026-08

用执行质量分数提升机器人策略训练效率,减少无效试错。

Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups

论文配图:Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups
图 1 · 摘自论文原文
  • 引入轨迹质量评分,将全胜或全败组拆分为质量梯度,恢复训练信号。
  • 在相同采样预算下,成功率最高提升56%,且避免奖励作弊行为。
  • 适用于仿真到真实机器人的迁移,适合强化学习与机器人控制研究者。

GRPO 因无需训练价值函数而被广泛用于视觉-语言-动作(VLA)策略的强化学习。但其需对每场景进行多次推演,代价高昂。在二值成功奖励下,所有推演均成功或失败的组别优势为零,会被动态采样丢弃。这类组别在训练早期尤为常见,导致大量昂贵的机器人推演被浪费。本文提出 Prism-GRPO,通过引入加权轨迹级执行质量评分,将同结果组按质量分层,恢复训练信号,同时保证所有成功仍优于所有失败。质量评分可来自模拟器接触、执行动作或视觉观察,无需任务特定进度奖励。理论证明:Prism-GRPO 不增加组别被丢弃的概率,并在满足梯度对齐条件下,更新仍为任务成功率的局部上升方向。在四个涵盖不同时间跨度与协作模式的 RoboTwin 任务中,Prism-GRPO 在匹配推演预算下显著提升成功率与质量,达成目标成功率所需推演次数最多减少56%。消融实验表明,接触、平滑性及基于 VLM 的质量信号均带来稳定增益。该方法还抑制了奖励劫持陷阱,其更优行为可直接迁移到真实机器人上部署。

原文摘要 · Abstract (English)

GRPO is increasingly used for reinforcement learning of vision-language-action (VLA) policies because, unlike PPO, it does not require training a critic. This simplification comes with a sampling cost: group-relative advantages require multiple rollouts from each scene. Under binary success rewards, groups whose rollouts all succeed or all fail have zero advantage and are discarded by dynamic sampling. These groups are especially common early in training, when most rollouts fail, wasting much of the expensive robotic rollout budget. We introduce Prism-GRPO, which augments binary outcome reward with a weighted trajectory-level execution-quality score. By splitting same-outcome groups into a quality spectrum, Prism-GRPO recovers training signal while ensuring that every success still outranks every failure. Quality scores can be derived from simulator contacts, executed actions, or visual observations, avoiding task-specific progress rewards. We prove that Prism-GRPO never increases the probability that a sampled group is discarded for having zero advantages, and derive a gradient-alignment condition under which its combined update remains a local ascent direction for task success. Across four RoboTwin tasks spanning different horizons and coordination patterns, Prism-GRPO improves success and quality at matched rollout budgets and reaches target success rates with up to 56% fewer rollouts. It also suppresses a reward-hacking shortcut, with the cleaner behavior transferring under direct deployment to a real robot. Through ablations, we show consistent gains across contact-, smoothness-, and VLM-derived quality signals.

强化学习机器人控制策略优化仿真迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。