用强化学习微调视觉语言模型,让机器人持续学新任务且数据更少。
Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning
- 通过分段策略的强化学习,自动评估动作片段质量。
- 在LIBERO上比传统方法成功率高22%,仅需20%数据。
- 适合需要长期学习新技能的机器人系统开发。
预训练的视觉-语言-动作(VLA)模型在多种任务中展现出强大的泛化能力,但主流的监督微调(SFT)方法依赖大量特定任务数据,易引发灾难性遗忘。为此,我们提出LifeLong-RFT——一种无需在线环境反馈或预训练奖励模型的强化微调策略。该方法结合分段级在线强化学习与多维度过程奖励机制,从三个维度量化中间动作片段的贡献:(1)离散动作一致性奖励(QACR)确保动作空间内预测准确;(2)连续轨迹对齐奖励(CTAR)使解码动作与参考轨迹一致,提升控制精度;(3)格式合规奖励(FCR)保障输出结构正确。在SimplerEnv、LIBERO及真实世界任务上的实验表明,LifeLong-RFT在多任务学习中表现优异。尤其在持续学习场景下,相比SFT,其平均成功率提升22%,且仅需20%训练数据即可有效适应新任务。该方法为VLA模型提供了一种有前景的后训练范式。
原文摘要 · Abstract (English)
Pretrained on large-scale and diverse datasets, VLA models demonstrate strong generalization and adaptability as general-purpose robotic policies. However, Supervised Fine-Tuning (SFT), which serves as the primary mechanism for adapting VLAs to downstream domains, requires substantial amounts of task-specific data and is prone to catastrophic forgetting. To address these limitations, we propose LifeLong-RFT, a simple yet effective Reinforcement Fine-Tuning (RFT) strategy for VLA models independent of online environmental feedback and pre-trained reward models. By integrating chunking-level on-policy reinforcement learning with the proposed multi-dimensional process reward mechanism, LifeLong-RFT quantifies the heterogeneous contributions of intermediate action chunks across three dimensions to facilitate policy optimization. Specifically, (1) the Quantized Action Consistency Reward (QACR) ensures accurate action prediction within the discrete action space; (2) the Continuous Trajectory Alignment Reward (CTAR) aligns decoded continuous action chunks with reference trajectories to ensure precise control; (3) the Format Compliance Reward (FCR) guarantees the structural validity of outputs. Comprehensive experiments across SimplerEnv, LIBERO, and real-world tasks demonstrate that LifeLong-RFT exhibits strong performance in multi-task learning. Furthermore, for continual learning on the LIBERO benchmark, our method achieves a 22% gain in average success rate over SFT, while effectively adapting to new tasks using only 20% of the training data. Overall, our method provides a promising post-training paradigm for VLAs. The project page is available at <https://yuan-liu-lifelong-rft.github.io>.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。