实时理解动态3D场景，支持在线推理与增量几何信息注入。
Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors
- 用自回归流控机制决定何时响应，结合轻量级空间特征融合模块。
- 在29项任务上超越主流模型，100万条在线时空问答对支撑评测。
- 适合需要实时3D感知的机器人、自动驾驶等应用开发者。
尽管3D场景理解取得进展，现有3D大模型仍局限于离线设置，需完整场景或预定义视频片段。本文提出Stream3D-VLM，一种支持从流式视频中实现实时空间理解的在线3D视觉-语言模型。方法基于大语言模型的下一个词预测目标，采用自回归流控建模以学习响应时机，并引入轻量级视觉-空间特征融合（VSFI）模块，将时间对齐的几何先验逐步注入视觉流。为缓解长上下文解码开销，提出即插即用的几何自适应体素压缩（GAVC）模块实现高效视觉标记压缩。针对流式3D-语言数据稀缺问题，构建可扩展的数据生成管道，创建超过100万条在线时空3D问答对，并建立覆盖29项任务的综合基准。大量实验表明，该方法在在线与离线3D空间理解、推理与定位任务中显著优于各类专有及开源模型。
原文摘要 · Abstract (English)
Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction objective to learn when to respond, and employs a lightweight Visual-Spatial Feature Integration (VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M online spatio-temporal 3D QA pairs and establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。