用视频生成模型实现自动驾驶感知到动作的端到端闭环。
VaViM and VaVAM: Autonomous Driving through Video Generative Modeling
- 基于自回归视频令牌序列预测帧，捕捉驾驶场景语义与动态
- 通过模仿学习从视频表征生成驾驶轨迹，闭环测试展现潜力
- 适合研究生成式自动驾驶与大规模视觉预训练的从业者
我们探索了大规模生成式视频模型在自动驾驶中的应用，提出一个开源的自回归视频模型VaViM及其配套的视频-动作模型VaVAM，以研究视频预训练如何迁移到真实驾驶场景。VaViM是一种简单的自回归视频模型，通过时空令牌序列预测帧，能够捕捉驾驶场景的语义和动态。VaVAM利用VaViM学习到的表征，通过模仿学习生成驾驶轨迹。两者共同构成完整的感知到动作的端到端流程。我们在开放和闭包环路驾驶场景中评估模型，结果表明基于视频的预训练在自动驾驶中具有前景。关键发现包括：学习表征的语义丰富性、视频合成中规模扩展的优势，以及模型规模、数据量与安全指标之间复杂的相互关系。代码与模型权重已公开于https://github.com/valeoai/VideoActionModel。
原文摘要 · Abstract (English)
We explore the potential of large-scale generative video models for autonomous driving, introducing an open-source auto-regressive video model (VaViM) and its companion video-action model (VaVAM) to investigate how video pre-training transfers to real-world driving. VaViM is a simple auto-regressive video model that predicts frames using spatio-temporal token sequences. We show that it captures the semantics and dynamics of driving scenes. VaVAM, the video-action model, leverages the learned representations of VaViM to generate driving trajectories through imitation learning. Together, the models form a complete perception-to-action pipeline. We evaluate our models in open- and closed-loop driving scenarios, revealing that video-based pre-training holds promise for autonomous driving. Key insights include the semantic richness of the learned representations, the benefits of scaling for video synthesis, and the complex relationship between model size, data, and safety metrics in closed-loop evaluations. We release code and model weights at https://github.com/valeoai/VideoActionModel
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。