用大世界模型实现闭环自动驾驶，统一感知预测规划
Doe-1: Closed-Loop Autonomous Driving with Large World Model
- 将驾驶任务建模为多模态令牌的自回归生成
- 在nuScenes上实现视觉问答、视频生成与路径规划
- 适合研究端到端自动驾驶与世界模型的学者
端到端自动驾驶因可从海量数据中学习而备受关注，但现有方法多为开环，存在扩展性差、高阶交互弱、决策效率低等问题。本文提出闭环自动驾驶框架Doe-1，基于大型驾驶世界模型（Driving World Model）实现感知、预测与规划的统一。将自动驾驶建模为下一个令牌生成问题，采用多模态令牌完成不同任务：用自由文本（场景描述）进行感知，以图像令牌在RGB空间直接生成未来预测；规划阶段使用位置感知分词器将动作编码为离散令牌。通过多模态Transformer端到端联合生成感知、预测与规划令牌。在nuScenes数据集上的实验表明，Doe-1在视觉问答、动作条件视频生成及运动规划等任务中均具有效性。
原文摘要 · Abstract (English)
End-to-end autonomous driving has received increasing attention due to its potential to learn from large amounts of data. However, most existing methods are still open-loop and suffer from weak scalability, lack of high-order interactions, and inefficient decision-making. In this paper, we explore a closed-loop framework for autonomous driving and propose a large Driving wOrld modEl (Doe-1) for unified perception, prediction, and planning. We formulate autonomous driving as a next-token generation problem and use multi-modal tokens to accomplish different tasks. Specifically, we use free-form texts (i.e., scene descriptions) for perception and generate future predictions directly in the RGB space with image tokens. For planning, we employ a position-aware tokenizer to effectively encode action into discrete tokens. We train a multi-modal transformer to autoregressively generate perception, prediction, and planning tokens in an end-to-end and unified manner. Experiments on the widely used nuScenes dataset demonstrate the effectiveness of Doe-1 in various tasks including visual question-answering, action-conditioned video generation, and motion planning. Code: https://github.com/wzzheng/Doe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。