arXiv:2512.23180cs.CV2025-12中稿 · CVPR被引 28

用3D高斯表示构建统一驾驶世界模型,实现环境理解与多模态生成。

GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation

  • 基于3D高斯原语嵌入语言特征,实现文本与场景早期对齐
  • 在nuScenes和NuInteract上达最新性能,支持多模态生成
  • 适合自动驾驶、具身智能等需场景理解的领域

驾驶世界模型(DWMs)随生成模型发展迅速,但现有方法缺乏3D场景理解能力,仅能基于输入生成内容,无法解释或推理驾驶环境。当前方法使用点云或鸟瞰图特征表示3D空间,难以准确对齐文本信息与底层3D场景。为此,我们提出一种基于3D高斯场景表示的统一DWM框架,同时实现3D场景理解与多模态场景生成,并支持理解与生成任务的上下文增强。通过将丰富的语言特征嵌入每个高斯原语,直接实现文本与3D场景的对齐。设计任务感知的语言引导采样策略,移除冗余3D高斯并注入精确紧凑的3D标记至大语言模型。此外,构建双条件多模态生成模型,利用视觉-语言模型捕获的信息作为高层语言条件,结合低层图像条件,共同引导多模态生成过程。我们在nuScenes和NuInteract数据集上进行综合性验证,结果表明该框架性能达到最新水平。代码将公开于GitHub:https://github.com/dtc111111/GaussianDWM。

原文摘要 · Abstract (English)

Driving World Models (DWMs) have been developing rapidly with the advances of generative models. However, existing DWMs lack 3D scene understanding capabilities and can only generate content conditioned on input data, without the ability to interpret or reason about the driving environment. Moreover, current approaches represent 3D spatial information with point cloud or BEV features do not accurately align textual information with the underlying 3D scene. To address these limitations, we propose a novel unified DWM framework based on 3D Gaussian scene representation, which enables both 3D scene understanding and multi-modal scene generation, while also enabling contextual enrichment for understanding and generation tasks. Our approach directly aligns textual information with the 3D scene by embedding rich linguistic features into each Gaussian primitive, thereby achieving early modality alignment. In addition, we design a novel task-aware language-guided sampling strategy that removes redundant 3D Gaussians and injects accurate and compact 3D tokens into LLM. Furthermore, we design a dual-condition multi-modal generation model, where the information captured by our vision-language model is leveraged as a high-level language condition in combination with a low-level image condition, jointly guiding the multi-modal generation process. We conduct comprehensive studies on the nuScenes, and NuInteract datasets to validate the effectiveness of our framework. Our method achieves state-of-the-art performance. We will release the code publicly on GitHub https://github.com/dtc111111/GaussianDWM.

3D高斯驾驶世界模型多模态生成场景理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。