arXiv:2607.08974cs.ROcs.AI2026-07

用语言动作描述引导动作生成,让视觉语言模型直接变智能机器人控制器。

CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

论文配图:CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding
图 1 · 摘自论文原文
  • 给动作序列前加自然语言描述,让模型按计划生成精确动作。
  • 20亿参数模型在LIBERO上达90.8%准确率,比基线高14.9个百分点。
  • 仅一次微调即可实现,适合研究视觉语言模型如何迁移到控制任务。

视觉-语言-动作模型(VLAs)继承自预训练的视觉语言模型(VLMs)的语义能力,但大规模机器人数据微调和架构修改会显著改变骨干网络,使难以区分VLM的实际贡献。通过最小架构改动将预训练VLM直接转化为VLA,能更透明地理解能力迁移过程。核心障碍在于输出分布不匹配:将动作作为纯数值标记序列预测,偏离了VLM预训练的语言分布,导致期望能力下降。为此,我们提出CLAP(因果语言-动作预测),在每个数值动作序列前添加自然语言动作描述,通过因果条件使精确动作标记预测依赖于语言-动作规划,且无需修改骨干结构。仅单轮微调下,20亿参数的CLAP在LIBERO上达到90.8%准确率(较VLA-0提升14.9个百分点),并在语言、物体和空间扰动下提升鲁棒性。我们将发布0.8亿、20亿和40亿参数的CLAP模型,作为单一VLM谱系的开源多尺度紧凑型VLA家族,支持对VLM到VLA能力迁移的可控分析。

原文摘要 · Abstract (English)

Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control. Directly converting pretrained VLMs into VLAs with minimal architectural change offers a more transparent path to understanding how VLM capabilities transfer across model scales. The core obstacle is output-distribution mismatch: predicting actions as bare numeric token sequences moves generation away from the VLM's pretrained language distribution, degrading the capabilities we seek to preserve. To address this, we propose CLAP (Causal Language-Action Prediction), which prepends each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture. With single-epoch fine-tuning alone, 2B CLAP achieves 90.8% on LIBERO (+14.9 pt over VLA-0) and improves robustness on LIBERO-PRO under language, object, and spatial perturbations. We will release CLAP at 0.8B, 2B, and 4B as an open-weight, multi-scale compact VLA family from a single VLM lineage, enabling controlled analysis of VLM-to-VLA capability transfer.

视觉语言机器人控制模型迁移动作生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。