FORCE让机器人视觉语言模型更高效地通过强化学习提升表现。
FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
- 用价值校准预热稳定初始训练，避免模型崩溃。
- 成功率达79%提升，比之前方法高10%，训练快32.5%。
- 无需人工干预，适合自主机器人部署场景。
视觉-语言-动作（VLA）模型常受限于低质量数据带来的模仿上限。虽然强化学习（RL）微调可突破此限制，但其样本效率极低。问题源于两个核心：（1）不稳定的Q函数导致灾难性初始遗忘；（2）低质量探索数据造成策略更新效率低下，常需昂贵的人工干预。本文提出FORCE，一种三阶段框架，通过解决这两个问题实现训练稳定。首先，引入价值校准预热阶段，利用在线回溯降低Q函数的分布偏移。随后，在在线阶段，该校准后的Q函数作为过滤器，仅允许高价值动作用于策略更新，涵盖自生成动作与专家数据。我们在多种仿真和真实任务上评估FORCE，结果表明其成功率绝对提升79%，优于先前RL方法10%，同时训练速度加快32.5%。关键的是，它缓解了常见成功率下降问题，且无需人工干预，为部署高效自主机器人代理迈出重要一步。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。