用虚拟世界模型替代真实环境,让机器人模型仅靠5次示范就能高效学习复杂操作。
World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training
- 用物理一致的虚拟环境代替真实交互,安全低成本地训练机器人。
- 仅需5次专家示范,就能显著提升机器人在复杂任务中的成功率。
- 适合工业自动化等高风险场景,解决真实环境不可复位难题。
通过模仿学习训练的视觉-语言-动作(VLA)模型在数据稀缺场景下性能严重下降,因其依赖大规模示范数据。尽管强化学习(RL)后训练有效缓解数据稀缺问题,但其应用受限于真实环境无法重置,尤其在工业自动化等高风险领域,交互常导致难以恢复的状态变化。此外,现有方法缺乏可靠的任务完成检测机制,导致冗余动作降低任务成功率。为此,我们提出 World-Env,一种基于强化学习的后训练框架,以低成本的世界模型虚拟模拟器替代物理交互。World-Env 包含两个关键组件:(1) 物理一致性世界模拟器,生成时间上连贯的未来视觉观测;(2) 由视觉-语言模型引导的即时反射器,提供持续奖励信号并预测动作终止。该模拟环境使 VLA 模型能安全探索并超越初始模仿学习分布。实验表明,该方法在复杂机器人操作任务中,仅需每任务5次专家示范即实现显著性能提升,有效克服传统 VLA 模型的数据效率低、安全约束强和执行效率差的问题,为资源受限环境下的后训练提供实用且可扩展的解决方案。代码已开源:https://github.com/amap-cvlab/world-env。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models trained via imitation learning suffer from significant performance degradation in data-scarce scenarios due to their reliance on large-scale demonstration datasets. Although reinforcement learning (RL)-based post-training has proven effective in addressing data scarcity, its application to VLA models is hindered by the non-resettable nature of real-world environments. This limitation is particularly critical in high-risk domains such as industrial automation, where interactions often induce state changes that are costly or infeasible to revert. Furthermore, existing VLA approaches lack a reliable mechanism for detecting task completion, leading to redundant actions that reduce overall task success rates. To address these challenges, we propose World-Env, an RL-based post-training framework that replaces physical interaction with a low-cost world model-based virtual simulator. World-Env consists of two key components: (1) a physically-consistent world simulator that generates temporally consistent future visual observations, and (2) a vision-language model (VLM)-guided instant reflector that provides continuous reward signals and predicts action termination. This simulated environment enables VLA models to safely explore and generalize beyond their initial imitation learning distribution. Our method achieves notable performance gains with as few as five expert demonstrations per task. Experiments on complex robotic manipulation tasks demonstrate that World-Env effectively overcomes the data inefficiency, safety constraints, and inefficient execution of conventional VLA models that rely on real-world interaction, offering a practical and scalable solution for post-training in resource-constrained settings. Our code is available at https://github.com/amap-cvlab/world-env.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。