用GPT思路构建自动驾驶世界模型,生成超40秒高保真视频。
DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
- 融合时空机制,同时捕捉帧间连续性和帧内空间信息。
- 生成长达40秒以上视频,比现有模型长2倍以上。
- 新掩码和重加权策略提升长期生成稳定性,适合可控场景模拟。
自回归生成模型(如GPT系列)在自然语言处理中取得成功,推动了视觉任务中的应用探索。一些研究尝试构建基于视频的世界模型以生成未来视频序列并预测车辆状态,但以往方法因经典GPT框架仅擅长处理一维上下文(如文本),缺乏对视频所需时空动态的建模能力,导致效果不佳。本文提出DrivingWorld,一种面向自动驾驶的GPT风格世界模型,引入多种时空融合机制,有效建模空间与时间动态,实现高质量、长时序视频生成。具体地,采用下一状态预测策略建模连续帧间的时序一致性,结合下一标记预测策略捕获单帧内的空间特征。为增强泛化能力,设计新颖的掩码策略与重加权机制,缓解长期生成中的漂移问题,实现精准控制。实验表明,该方法可生成超过40秒的高保真视频,时长超过当前最先进模型两倍以上;视觉质量更优,可控未来视频生成精度显著提升。代码已公开于https://github.com/YvanYin/DrivingWorld。
原文摘要 · Abstract (English)
Recent successes in autoregressive (AR) generation models, such as the GPT series in natural language processing, have motivated efforts to replicate this success in visual tasks. Some works attempt to extend this approach to autonomous driving by building video-based world models capable of generating realistic future video sequences and predicting ego states. However, prior works tend to produce unsatisfactory results, as the classic GPT framework is designed to handle 1D contextual information, such as text, and lacks the inherent ability to model the spatial and temporal dynamics essential for video generation. In this paper, we present DrivingWorld, a GPT-style world model for autonomous driving, featuring several spatial-temporal fusion mechanisms. This design enables effective modeling of both spatial and temporal dynamics, facilitating high-fidelity, long-duration video generation. Specifically, we propose a next-state prediction strategy to model temporal coherence between consecutive frames and apply a next-token prediction strategy to capture spatial information within each frame. To further enhance generalization ability, we propose a novel masking strategy and reweighting strategy for token prediction to mitigate long-term drifting issues and enable precise control. Our work demonstrates the ability to produce high-fidelity and consistent video clips of over 40 seconds in duration, which is over 2 times longer than state-of-the-art driving world models. Experiments show that, in contrast to prior works, our method achieves superior visual quality and significantly more accurate controllable future video generation. Our code is available at https://github.com/YvanYin/DrivingWorld.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。