用多模态自回归模型统一驾驶世界建模与路径规划
DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers
- 将图像和动作令牌交错编码，构建统一的驾驶语言
- 在nuPlan和NAVSIM上实现超越基线的生成与规划性能
- 适合自动驾驶系统研发者及多模态序列建模研究者
基于世界模型的搜索与规划被广泛认为是实现人类级物理智能的可行路径。然而，当前驾驶世界模型主要依赖视频扩散模型，擅长视觉生成但难以融合动作等其他模态。相比之下，自回归变压器在建模多模态数据方面表现卓越。本文旨在将驾驶模型仿真与轨迹规划统一为单一序列建模问题。我们提出一种基于图像与动作令牌交错的多模态驾驶语言，并开发DrivingGPT，通过标准的下一个标记预测学习联合世界建模与规划。DrivingGPT在动作条件下的视频生成与端到端规划任务中均表现出色，在大规模nuPlan和NAVSIM基准上优于多个强基线。
原文摘要 · Abstract (English)
World model-based searching and planning are widely recognized as a promising path toward human-level physical intelligence. However, current driving world models primarily rely on video diffusion models, which specialize in visual generation but lack the flexibility to incorporate other modalities like action. In contrast, autoregressive transformers have demonstrated exceptional capability in modeling multimodal data. Our work aims to unify both driving model simulation and trajectory planning into a single sequence modeling problem. We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning through standard next-token prediction. Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。