统一扩散模型同步生成图像与动作,提升机器人任务执行效率。
Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process
- 用联合去噪过程同步处理视觉、语言和动作,实现端到端协同。
- 在CALVIN等数据集上达到顶尖性能,推理速度比自回归方法快4倍。
- 适合需要高效多模态决策的机器人应用,如家庭服务与工业自动化。
视觉-语言-动作(VLA)模型旨在理解自然语言指令和视觉观察,并作为具身智能体执行相应动作。近期工作将未来图像纳入理解-执行循环,构建了统一的VLA模型,能同时理解、生成和行动——读取文本与图像并输出未来图像和动作。然而,这些模型要么依赖外部专家进行模态统一,要么将图像生成与动作预测分开处理,限制了两任务间的直接协同优势。本文核心思想是通过同步去噪过程,联合优化生成与动作,使动作在持续且充分的视觉引导下从初始状态逐步演化。我们提出统一扩散VLA和联合离散去噪扩散过程(JD3P),这是一个将多种模态融合于单一去噪轨迹的联合扩散过程,成为理解、生成与行动内在协同的关键机制。模型基于所有模态的统一标记空间和混合注意力机制构建,并设计了两阶段训练流程与多个推理时优化技术。在CALVIN、LIBERO和SimplerEnv等基准上达到最先进水平,推理速度比自回归方法快4倍,并通过深入分析与真实世界评估验证有效性。
原文摘要 · Abstract (English)
Vision-language-action (VLA) models aim to understand natural language instructions and visual observations and to execute corresponding actions as an embodied agent. Recent work integrates future images into the understanding-acting loop, yielding unified VLAs that jointly understand, generate, and act -- reading text and images and producing future images and actions. However, these models either rely on external experts for modality unification or treat image generation and action prediction as separate processes, limiting the benefits of direct synergy between these tasks. Our core philosophy is to optimize generation and action jointly through a synchronous denoising process, where the iterative refinement enables actions to evolve from initialization, under constant and sufficient visual guidance. We ground this philosophy in our proposed Unified Diffusion VLA and Joint Discrete Denoising Diffusion Process (JD3P), which is a joint diffusion process that integrates multiple modalities into a single denoising trajectory to serve as the key mechanism enabling understanding, generation, and acting to be intrinsically synergistic. Our model and theory are built on a unified tokenized space of all modalities and a hybrid attention mechanism. We further propose a two-stage training pipeline and several inference-time techniques that optimize performance and efficiency. Our approach achieves state-of-the-art performance on benchmarks such as CALVIN, LIBERO, and SimplerEnv with 4$\times$ faster inference than autoregressive methods, and we demonstrate its effectiveness through in-depth analysis and real-world evaluations. Our project page is available at https://irpn-eai.github.io/UD-VLA.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。