arXiv:2512.05060cs.CV2025-12被引 4

首个统一的4D语言-视觉几何Transformer,实现动态场景的高效语义理解。

4DLangVGGT: 4D Language-Visual Geometry Grounded Transformer

  • 用Transformer架构联合建模时空几何与语言语义,无需每场景优化。
  • 在多场景数据上训练,推理时直接应用,提升部署效率和泛化能力。
  • 在HyperNeRF和Neu3D数据集上达领先性能,多场景训练提升1%。

构建4D语言场对具身智能、增强/虚拟现实及4D场景理解至关重要,能提供动态环境的丰富语义表示并支持开放词汇查询。然而现有方法主要依赖场景特定的高斯泼溅,需逐场景优化,泛化性差且难以规模化。为此,我们提出首个基于Transformer的4D语言-视觉几何统一框架4DLangVGGT,其包含4D视觉几何变压器StreamVGGT,用于捕捉动态场景的时空几何特征;以及语义桥接解码器SBD,将几何感知特征投影至语言对齐语义空间,增强可解释性同时保持结构保真度。不同于依赖昂贵场景优化的方法,4DLangVGGT可在多个动态场景上联合训练,并直接用于推理,显著提升大规模部署实用性,确立开放词汇4D场景理解新范式。在HyperNeRF与Neu3D数据集上的实验表明,该方法不仅具有良好泛化性,还达到最先进水平,在单场景训练下最高提升2%,多场景训练下提升1%。代码已开源。

原文摘要 · Abstract (English)

Constructing 4D language fields is crucial for embodied AI, augmented/virtual reality, and 4D scene understanding, as they provide enriched semantic representations of dynamic environments and enable open-vocabulary querying in complex scenarios. However, existing approaches to 4D semantic field construction primarily rely on scene-specific Gaussian splatting, which requires per-scene optimization, exhibits limited generalization, and is difficult to scale to real-world applications. To address these limitations, we propose 4DLangVGGT, the first Transformer-based feed-forward unified framework for 4D language grounding, that jointly integrates geometric perception and language alignment within a single architecture. 4DLangVGGT has two key components: the 4D Visual Geometry Transformer, StreamVGGT, which captures spatio-temporal geometric representations of dynamic scenes; and the Semantic Bridging Decoder (SBD), which projects geometry-aware features into a language-aligned semantic space, thereby enhancing semantic interpretability while preserving structural fidelity. Unlike prior methods that depend on costly per-scene optimization, 4DLangVGGT can be jointly trained across multiple dynamic scenes and directly applied during inference, achieving both deployment efficiency and strong generalization. This design significantly improves the practicality of large-scale deployment and establishes a new paradigm for open-vocabulary 4D scene understanding. Experiments on HyperNeRF and Neu3D datasets demonstrate that our approach not only generalizes effectively but also achieves state-of-the-art performance, achieving up to 2% gains under per-scene training and 1% improvements under multi-scene training. Our code released in https://github.com/hustvl/4DLangVGGT

4D生成视觉语言Transformer场景理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。