用模型自生成成功轨迹做参考,让机器人在少样本下快速学会复杂操作。
SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
- 利用当前训练批次内的成功轨迹作为自我参照,生成进度奖励
- 200步强化学习后在LIBERO上达到99.2%成功率,相对提升103%
- 无需人工标注或额外数据,适合低资源场景下的机器人训练
视觉-语言-动作(VLA)模型在机器人操作中表现优异,但严重依赖专家示范,导致示范偏差并限制性能。强化学习是突破此限制的关键策略,但现有VLA-RL方法受限于严重的奖励稀疏性。依赖二值成功信号会浪费失败轨迹中的宝贵信息,导致训练效率低下。为此,本文提出自参照策略优化(SRPO),一种新型VLA-RL框架。SRPO通过利用当前训练批次内模型自身生成的成功轨迹作为自我参照,无需外部示范或人工奖励设计,即可为失败尝试分配进度奖励。核心创新在于使用世界模型的潜在空间表示来稳健度量行为进展,避免依赖原始像素或特定领域微调,其压缩且可迁移的编码能自然捕捉跨环境的进展模式,实现精准、泛化的轨迹对比。在LIBERO基准上的实证评估表明,SRPO具有显著的高效性与有效性:从48.9%的监督基线出发,仅用200步强化学习即达到99.2%的新高成功率,相对提升103%,且无额外监督;在LIBERO-Plus上更实现167%的性能提升。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations, leading to demonstration bias and limiting performance. Reinforcement learning (RL) is a vital post-training strategy to overcome these limits, yet current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity. Relying on binary success indicators wastes valuable information in failed trajectories, resulting in low training efficiency. To solve this, we propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework. SRPO eliminates the need for external demonstrations or manual reward engineering by leveraging the model's own successful trajectories, generated within the current training batch, as a self-reference. This allows us to assign a progress-wise reward to failed attempts. A core innovation is the use of latent world representations to measure behavioral progress robustly. Instead of relying on raw pixels or requiring domain-specific fine-tuning, we utilize the compressed, transferable encodings from a world model's latent space. These representations naturally capture progress patterns across environments, enabling accurate, generalized trajectory comparison. Empirical evaluations on the LIBERO benchmark demonstrate SRPO's efficiency and effectiveness. Starting from a supervised baseline with 48.9% success, SRPO achieves a new state-of-the-art success rate of 99.2% in just 200 RL steps, representing a 103% relative improvement without any extra supervision. Furthermore, SRPO shows substantial robustness, achieving a 167% performance improvement on the LIBERO-Plus benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。