用视觉语言模型指导机器人学习,让训练更快更省样本。
Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents

- 让大模型提供高层动作建议,引导强化学习早期探索。
- 减少一半以上环境交互次数,真实机器人也能零样本迁移。
- 适合需要快速学新任务的机器人系统,尤其擅长复杂操作。
强化学习可实现机器人的高频闭环控制,但面对长时程、奖励稀疏或不完整的任务时,探索效率低且信用分配差。视觉-语言-动作(VLA)模型通过大规模多模态预训练具备通用任务推理能力,但现有方法难以直接用于高速精准操作。本文提出视觉-语言-动作跳启动(VLAJS),将稀疏的VLA指导与在线强化学习结合,提升探索与学习效率。VLAJS将VLA视为临时的高层动作建议源,在训练初期引导探索并改善信用分配,同时保留强化学习的高频状态控制。方法在近端策略优化(PPO)基础上引入方向性动作一致性正则化,软性对齐动作与VLA建议,无需严格模仿、演示或持续教师查询。VLA指导稀疏应用并随时间衰减,使智能体能在线适应并最终超越引导策略。我们在六个模拟任务(提升、拾取放置、插销旋转、插销插入、戳动、推动)和一个真实Franka Panda机器人上验证。VLAJS在样本效率上显著优于PPO及蒸馏基线,在多个任务中减少超过50%环境交互;真实实验表明其具备零样本仿真到现实迁移能力,能在杂乱、物体变化及外部扰动下稳定执行。
原文摘要 · Abstract (English)
Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation, but scaling to long-horizon tasks with sparse or imperfect rewards remains difficult due to inefficient exploration and poor credit assignment. Vision-Language-Action (VLA) models leverage large-scale multimodal pretraining to provide generalist, task-level reasoning, but current limitations hinder their direct use in fast and precise manipulation. In this paper, we propose Vision-Language-Action Jump-Starting (VLAJS), a method that bridges sparse VLA guidance with on-policy RL to improve exploration and learning efficiency. VLAJS treats VLAs as transient sources of high-level action suggestions that bias early exploration and improve credit assignment, while preserving the high-frequency, state-based control of RL. Our approach augments Proximal Policy Optimization (PPO) with a directional action-consistency regularization that softly aligns the RL agent's actions with VLA guidance during early training, without enforcing strict imitation, requiring demonstrations, or relying on continuous teacher queries. VLA guidance is applied sparsely and annealed over time, allowing the agent to adapt online and ultimately surpass the guiding policy. We evaluate VLAJS on six challenging manipulation tasks: lifting, pick-and-place, peg reorientation, peg insertion, poking, and pushing in simulation, and validate a subset on a real Franka Panda robot. VLAJS consistently outperforms PPO and distillation-style baselines in sample efficiency, reducing required environment interactions by over 50% in several tasks. Real-world experiments demonstrate zero-shot sim-to-real transfer and robust execution under clutter, object variation, and external perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。