用动作分块PPO+自收集示范，让视觉语言动作模型训练更稳定高效。
VLA Model Post-Training via Action-Chunked PPO and Self Behavior Cloning
- 将连续动作分块处理，提升策略时间一致性与反馈密度。
- 在MetaWorld上达成0.93成功率，平均42.17步完成任务。
- 适合需要高可靠性的机器人控制类VLA应用开发者。
强化学习（RL）是后训练视觉-语言-动作（VLA）模型的有前景方向，但稀疏奖励和训练不稳定限制了实际部署。本文提出基于动作分块的近端策略优化（PPO）方法，结合自收集示范的行为克隆。将连续动作聚合为动作块，提升了策略的时间一致性与信息反馈密度。同时引入辅助行为克隆损失，使用动态更新的示范缓冲区持续收集高质量任务样本。在线调整动作分块PPO与自行为克隆损失的相对权重，以稳定后训练过程。在MetaWorld基准测试中，性能优于监督微调，实现0.93的成功率和42.17步的平均完成步数。结果证明了强化学习在VLA后训练中的可行性，为下游VLA应用奠定了基础。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a promising avenue for post-training vision-language-action (VLA) models, but practical deployment is hindered by sparse rewards and unstable training. This work mitigates these challenges by introducing an action chunk based on proximal policy optimization (PPO) with behavior cloning using self-collected demonstrations. Aggregating consecutive actions into chunks improves the temporal consistency of the policy and the density of informative feedback. In addition, an auxiliary behavior cloning loss is applied with a dynamically updated demonstration buffer that continually collects high-quality task trials during training. The relative weight between the action-chunked PPO objective and the self behavior clone auxiliary loss is adapted online to stabilize the post-training process. Experiments on the MetaWorld benchmark indicate improved performance over supervised fine-tuning, achieving a high success rate (0.93) and few steps to success (42.17). These results demonstrate the viability of RL for VLA post-training and help lay the groundwork for downstream VLA applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。