arXiv:2605.28544cs.CV2026-05被引 7

用视频生成模型构建自动驾驶的时空动作策略,实现长时序规划。

DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving

论文配图:DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving
图 1 · 摘自论文原文
  • 将视频扩散模型改造成自回归动作生成器,统一处理视频与动作序列
  • 在两个基准上实现强规划性能,100k数据训练后效果显著提升
  • 适合研究端到端自动驾驶、视频生成与长时序决策的学者

预训练基础模型已成为端到端自动驾驶的重要基础。与主要基于静态图像-文本对训练的视觉语言模型不同,视频生成模型能捕捉时间动态和运动先验,更契合驾驶任务。我们提出DriveWAM,一种将预训练视频扩散变压器适配为自回归视频-动作策略的驾驶世界-动作模型。DriveWAM将视频与动作流组织成统一的时间标记序列,并在联合流匹配目标下进行训练,保留了预训练视频生成架构的同时,将其大规模视频先验迁移至动作生成。为引入高层场景理解,我们设计了场景演进驱动引导机制,利用冻结的视觉语言模型生成分块语义意图以指导视频-动作生成。为控制长时程推断的开销,进一步提出选择性键值记忆,通过相关性-冗余缓存选择机制,在推理时维持有界的模态感知视频与动作记忆池。在NAVSIM和PhysicalAI-Autonomous-Vehicles基准上的实验表明,DriveWAM表现出优异的规划性能;从4k到100k驾驶片段的数据缩放研究进一步验证了世界-动作建模在端到端自动驾驶中的可扩展潜力。

原文摘要 · Abstract (English)

Pretrained foundation models have become an important basis for end-to-end autonomous driving. In contrast to vision-language models pretrained primarily on static image-text pairs, video generative models capture temporal dynamics and motion priors that are naturally suited for driving. We present DriveWAM, a driving world-action model that adapts a pretrained video diffusion transformer into an autoregressive video-action policy. DriveWAM organizes video and action streams into a unified temporal token sequence and trains them under a joint flow-matching objective, preserving the pretrained video-generation architecture while adapting its large-scale video priors to action generation. To incorporate high-level scene understanding, we introduce scene-evolving driving guidance, where a frozen VLM produces chunk-specific semantic intent to guide video-action generation. To keep long-horizon rollout bounded, we further introduce selective KV memory, which maintains bounded modality-aware video and action memory pools through relevance-redundancy cache selection at inference time. Experiments on NAVSIM and the PhysicalAI-Autonomous-Vehicles benchmark show that DriveWAM achieves strong planning performance, and a data-scaling study from 4k to 100k driving clips further confirms the scaling potential of world-action modeling for end-to-end autonomous driving.

自动驾驶视频生成扩散模型长时序决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。