让机器人更懂动作背后的因果变化,提升决策效率与准确性。
$Δ$VLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variation
- 用先验知识引导提取当前环境关键信息,减少冗余。
- 通过离散潜变量建模世界状态变化,压缩预测维度。
- 注意力机制分离不同知识维度,避免干扰,适合真实机器人部署。
近期视觉-语言-动作(VLA)模型通过统一感知、推理与控制,显著推进了机器人操作能力。现有方法多采用预测未来视觉状态或世界知识的范式,但侧重结果预测,忽视变化过程的因果推理,而后者对行动决策至关重要。为此,我们提出ΔVLA,一种基于先验引导的世界知识变化建模框架,不直接回归绝对未来状态,而是建模相对于显式当前世界知识先验的变化。首先,设计先验引导的世界知识提取器(PWKE),利用辅助头和先验伪标签从视觉输入中提取可操作区域、空间关系与语义线索,降低冗余。其次,提出潜世界变化量化(LWVQ),通过VQ-VAE目标学习离散潜空间,编码世界知识变化,将预测从全模态压缩至紧凑潜变量。最后,设计条件变化注意力(CV-Atten),促进解耦学习,保持知识表示独立性。在仿真基准与真实机器人任务上的大量实验表明,ΔVLA达到顶尖性能且提升效率。代码与真实执行视频见https://github.com/JiuTian-VL/DeltaVLA。
原文摘要 · Abstract (English)
Recent vision-language-action (VLA) models have significantly advanced robotic manipulation by unifying perception, reasoning, and control. To achieve such integration, recent studies adopt a predictive paradigm that models future visual states or world knowledge to guide action generation. However, these models emphasize forecasting outcomes rather than reasoning about the underlying process of change, which is essential for determining how to act. To address this, we propose $Δ$VLA, a prior-guided framework that models world-knowledge variations relative to an explicit current-world knowledge prior for action generation, rather than regressing absolute future world states. Specifically, 1) to construct the current world knowledge prior, we propose the Prior-Guided WorldKnowledge Extractor (PWKE). It extracts manipulable regions, spatial relations, and semantic cues from the visual input, guided by auxiliary heads and prior pseudo labels, thus reducing redundancy. 2) Building upon this, to represent how world knowledge evolves under actions, we introduce the Latent World Variation Quantization (LWVQ). It learns a discrete latent space via a VQ-VAE objective to encode world knowledge variations, shifting prediction from full modalities to compact latent. 3)Moreover, to mitigate interference during variation modeling, we design the Conditional Variation Attention (CV-Atten), whichpromotes disentangled learning and preserves the independence of knowledge representations. Extensive experiments on both simulated benchmarks and real-world robotic tasks demonstrate $Δ$VLA achieves state-of-the-art performance while improving efficiency. Code and real-world execution videos are available at https://github.com/JiuTian-VL/DeltaVLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。