用强化学习优化视觉语言动作模型，让机器人学会更高效的任务执行。
Refined Policy Distillation: From VLA Generalists to RL Experts
- 结合强化学习与行为克隆，用VLA指导学生策略探索
- 在密集和稀疏奖励下均超越教师VLA性能，收敛更快
- 对视角变化和任务变体有良好泛化能力，适合机器人控制
视觉-语言-动作模型（VLAs）在真实世界实验中展现出优异的泛化能力，但其成功率常不及专家策略，且在环境变化时需重新微调。本文提出精炼策略蒸馏（RPD），一种基于强化学习的策略优化方法，通过结合在线策略强化学习与行为克隆，利用教师VLA的动作引导学生策略在强化学习中的探索过程，从而提升样本效率并加速收敛。我们在ManiSkill3仿真环境中对Octo和OpenVLA的微调版本进行了评估。实验结果表明，RPD使强化学习学生策略学习到的专家策略在密集与稀疏奖励设置下均优于原始教师VLA，且收敛速度超过基线。该方法对相机视角变化具有鲁棒性，并能泛化至原生VLA无法解决的任务变体。代码、数据集、VLA检查点及视频已公开于https://refined-policy-distillation.github.io
原文摘要 · Abstract (English)
Vision-Language-Action Models (VLAs) have demonstrated remarkable generalization capabilities in real-world experiments. However, their success rates are often not on par with expert policies, and they require fine-tuning when the setup changes. In this work, we introduce Refined Policy Distillation (RPD), a novel Reinforcement Learning (RL)-based policy refinement method that bridges this performance gap through a combination of on-policy RL with behavioral cloning. The core idea of RPD is to distill and refine VLAs into compact, high-performing expert policies by guiding the student policy during RL exploration using the actions of a teacher VLA, resulting in increased sample efficiency and faster convergence. We complement our method by fine-tuned versions of Octo and OpenVLA for ManiSkill3 to evaluate RPD in simulation. While this is a key requirement for applying RL, it also yields new insights beyond existing studies on VLA performance in real-world settings. Our experimental results across various manipulation tasks show that RPD enables the RL student to learn expert policies that outperform the VLA teacher in both dense and sparse reward settings, while also achieving faster convergence than the RL baseline. Our approach is even robust to changes in camera perspective and can generalize to task variations that the underlying VLA cannot solve. Our code, dataset, VLA checkpoints, and videos are available at https://refined-policy-distillation.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。