提升视觉语言动作模型的推理能力,让AI决策更像人一样逐步思考。
VLA-R1: Enhancing Reasoning in Vision-Language-Action Models
- 用可验证奖励强化推理过程,结合分组相对策略优化。
- 在多个场景中实现比现有方法更高的泛化与真实机器人表现。
- 提供带思维链标注的数据集,适合研究具身智能与决策推理。
视觉-语言-动作(VLA)模型旨在统一感知、语言理解与动作生成,在具身人工智能中展现出强大的跨任务与跨场景泛化能力。然而,当前VLA模型常缺乏显式的分步推理,直接输出最终动作,未考虑物体可用性或几何关系约束。其后训练流程也较少强化推理质量,主要依赖监督微调和弱奖励设计。为此,我们提出VLA-R1,一种增强推理能力的VLA模型,结合可验证奖励的强化学习(RLVR)与分组相对策略优化(GRPO),系统优化推理与执行。我们设计基于RLVR的后训练策略,利用可验证奖励对区域对齐、轨迹一致性与输出格式进行优化,从而提升推理鲁棒性与执行准确性。同时,构建高质量数据集VLA-CoT-13K,提供与可用性及轨迹标注对齐的思维链监督。大量实验在域内、域外、仿真与真实机器人平台均表明,VLA-R1显著优于现有VLA方法,在泛化与真实世界表现上更具优势。代码、模型与数据集将在论文发表后开源。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models aim to unify perception, language understanding, and action generation, offering strong cross-task and cross-scene generalization with broad impact on embodied AI. However, current VLA models often lack explicit step-by-step reasoning, instead emitting final actions without considering affordance constraints or geometric relations. Their post-training pipelines also rarely reinforce reasoning quality, relying primarily on supervised fine-tuning with weak reward design. To address these challenges, we present VLA-R1, a reasoning-enhanced VLA that integrates Reinforcement Learning from Verifiable Rewards (RLVR) with Group Relative Policy Optimization (GRPO) to systematically optimize both reasoning and execution. Specifically, we design an RLVR-based post-training strategy with verifiable rewards for region alignment, trajectory consistency, and output formatting, thereby strengthening reasoning robustness and execution accuracy. Moreover, we develop VLA-CoT-13K, a high-quality dataset that provides chain-of-thought supervision explicitly aligned with affordance and trajectory annotations. Furthermore, extensive evaluations on in-domain, out-of-domain, simulation, and real-robot platforms demonstrate that VLA-R1 achieves superior generalization and real-world performance compared to prior VLA methods. We plan to release the model, code, and dataset following the publication of this work. Code: https://github.com/GigaAI-research/VLA-R1. Website: https://gigaai-research.github.io/VLA-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。