用视频生成模型构建自动驾驶的时空动作策略,实现长时序规划。
DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving

- 将视频扩散模型改造成自回归动作生成器,统一处理视频与动作序列
- 在两个基准上实现强规划性能,100k数据训练后效果显著提升
- 适合研究端到端自动驾驶、视频生成与长时序决策的学者
预训练基础模型已成为端到端自动驾驶的重要基础。与主要基于静态图像-文本对训练的视觉语言模型不同,视频生成模型能捕捉时间动态和运动先验,更契合驾驶任务。我们提出DriveWAM,一种将预训练视频扩散变压器适配为自回归视频-动作策略的驾驶世界-动作模型。DriveWAM将视频与动作流组织成统一的时间标记序列,并在联合流匹配目标下进行训练,保留了预训练视频生成架构的同时,将其大规模视频先验迁移至动作生成。为引入高层场景理解,我们设计了场景演进驱动引导机制,利用冻结的视觉语言模型生成分块语义意图以指导视频-动作生成。为控制长时程推断的开销,进一步提出选择性键值记忆,通过相关性-冗余缓存选择机制,在推理时维持有界的模态感知视频与动作记忆池。在NAVSIM和PhysicalAI-Autonomous-Vehicles基准上的实验表明,DriveWAM表现出优异的规划性能;从4k到100k驾驶片段的数据缩放研究进一步验证了世界-动作建模在端到端自动驾驶中的可扩展潜力。
原文摘要 · Abstract (English)
Pretrained foundation models have become an important basis for end-to-end autonomous driving. In contrast to vision-language models pretrained primarily on static image-text pairs, video generative models capture temporal dynamics and motion priors that are naturally suited for driving. We present DriveWAM, a driving world-action model that adapts a pretrained video diffusion transformer into an autoregressive video-action policy. DriveWAM organizes video and action streams into a unified temporal token sequence and trains them under a joint flow-matching objective, preserving the pretrained video-generation architecture while adapting its large-scale video priors to action generation. To incorporate high-level scene understanding, we introduce scene-evolving driving guidance, where a frozen VLM produces chunk-specific semantic intent to guide video-action generation. To keep long-horizon rollout bounded, we further introduce selective KV memory, which maintains bounded modality-aware video and action memory pools through relevance-redundancy cache selection at inference time. Experiments on NAVSIM and the PhysicalAI-Autonomous-Vehicles benchmark show that DriveWAM achieves strong planning performance, and a data-scaling study from 4k to 100k driving clips further confirms the scaling potential of world-action modeling for end-to-end autonomous driving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。