构建多模态驾驶场景生成框架，支持长序列真实场景合成。
Generating Multimodal Driving Scenes via Next-Scene Prediction
- 四模态融合生成，引入地图模态并分阶段控制计算开销。
- 通过时序与有序自回归机制，实现跨帧与跨模态动态对齐。
- 动作感知地图对齐模块保障地图与车辆行为一致性，适合自动驾驶评估。
自动驾驶中的生成模型可创造多样场景，但现有方法仅覆盖有限模态，限制了对自动驾驶系统全面评估的能力。本文提出一种融合四种主要数据模态的多模态生成框架，首次引入地图模态。通过将各模态标记化，采用两阶段策略控制计算开销，实现场景序列的自回归预测。时序自回归（TAR）组件捕捉各模态的帧间动态，有序自回归（OAR）组件通过固定顺序逐个预测令牌，完成模态内对齐。为确保地图与自车行为的一致性，设计动作感知地图对齐（AMA）模块，基于自车动作施加变换以维持模态间连贯性。该框架能有效生成复杂且真实的长序列驾驶场景，保障多模态一致性，并提供对场景元素的细粒度控制。项目主页：https://yanhaowu.github.io/UMGen/
原文摘要 · Abstract (English)
Generative models in Autonomous Driving (AD) enable diverse scene creation, yet existing methods fall short by only capturing a limited range of modalities, restricting the capability of generating controllable scenes for comprehensive evaluation of AD systems. In this paper, we introduce a multimodal generation framework that incorporates four major data modalities, including a novel addition of map modality. With tokenized modalities, our scene sequence generation framework autoregressively predicts each scene while managing computational demands through a two-stage approach. The Temporal AutoRegressive (TAR) component captures inter-frame dynamics for each modality while the Ordered AutoRegressive (OAR) component aligns modalities within each scene by sequentially predicting tokens in a fixed order. To maintain coherence between map and ego-action modalities, we introduce the Action-aware Map Alignment (AMA) module, which applies a transformation based on the ego-action to maintain coherence between these modalities. Our framework effectively generates complex, realistic driving scenes over extended sequences, ensuring multimodal consistency and offering fine-grained control over scene elements. Project page: https://yanhaowu.github.io/UMGen/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。