实时视频中在线提取3D物体，兼顾几何与语义理解
EA3D: Online Open-World 3D Object Extraction from Streaming Videos
- 用视觉语言模型解析每一帧视频，动态更新高斯特征图
- 通过递归优化模块提升重建精度，支持多任务下游应用
- 适合需要实时3D场景理解的机器人、AR/VR等场景
现有3D场景理解方法受限于离线采集的多视角数据或预构建的3D几何结构。本文提出ExtractAnything3D（EA3D），一种统一的在线开放世界3D物体提取框架，实现几何重建与整体场景理解的同步进行。给定流式视频，EA3D利用视觉-语言和2D视觉基础编码器动态解析每一帧，提取对象级知识，并通过前馈式在线更新策略将其集成嵌入高斯特征图。随后，模型从历史帧中迭代估计视觉里程计，并根据新观测增量更新在线高斯特征。一个递归联合优化模块引导模型关注兴趣区域，同时增强几何重建与语义理解。在多个基准和任务上的实验表明，包括照片级渲染、语义与实例分割、3D边界框与语义占据估计、3D网格生成等，该方法表现出色。所提方法建立了一个统一高效的在线3D重建与整体场景理解框架，支持广泛的下游任务。
原文摘要 · Abstract (English)
Current 3D scene understanding methods are limited by offline-collected multi-view data or pre-constructed 3D geometry. In this paper, we present ExtractAnything3D (EA3D), a unified online framework for open-world 3D object extraction that enables simultaneous geometric reconstruction and holistic scene understanding. Given a streaming video, EA3D dynamically interprets each frame using vision-language and 2D vision foundation encoders to extract object-level knowledge. This knowledge is integrated and embedded into a Gaussian feature map via a feed-forward online update strategy. We then iteratively estimate visual odometry from historical frames and incrementally update online Gaussian features with new observations. A recurrent joint optimization module directs the model's attention to regions of interest, simultaneously enhancing both geometric reconstruction and semantic understanding. Extensive experiments across diverse benchmarks and tasks, including photo-realistic rendering, semantic and instance segmentation, 3D bounding box and semantic occupancy estimation, and 3D mesh generation, demonstrate the effectiveness of EA3D. Our method establishes a unified and efficient framework for joint online 3D reconstruction and holistic scene understanding, enabling a broad range of downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。