用深度引导的3D概率节点生成更稳定可靠的3D语义场景图
DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors
- 通过深度滤波估计实例级3D高斯分布，用概率节点替代点投影
- 在3DSSG和ReplicaSSG上三元组召回率提升77.4%，谓词召回率提升23.2%
- 适合机器人操作与增强现实应用，结构时序一致
我们提出DeWorldSG，一种从RGB-D序列生成时空鲁棒的3D语义场景图的新框架。现有方法常因不稳定的3D物体表征和帧间推理导致关系缺失，难以构建可靠3D场景图。DeWorldSG通过深度引导过滤估计实例级几何3D高斯分布，将每个物体表示为概率3D节点而非单一投影点。为缓解帧间推理造成的关联稀疏问题，该框架进一步跨对象对聚合时空证据，并利用世界模型（V-JEPA 2）提供的上下文先验来优化关系。在3DSSG和ReplicaSSG数据集上的实验表明，该方法在物体和谓词预测上均达到最新水平（SoTA），同时生成时序一致的场景结构。尤其相比先前SoTA方法，三元组召回率提升77.4%，谓词召回率提升23.2%，适用于机器人操作与增强现实任务。代码与模型已开源。
原文摘要 · Abstract (English)
We present DeWorldSG, a novel framework that generates spatio-temporally robust 3D Semantic Scene Graphs from RGB-D sequences. Existing methods often struggle to construct reliable 3D scene graphs due to unstable 3D object representations and missing relations caused by frame-wise inference. DeWorldSG addresses these issues by estimating instance-level geometric 3D Gaussian distributions through depth-guided filtering and representing each object as a probabilistic 3D node rather than a single projected point. To mitigate relational sparsity from frame-wise inference, our framework further aggregates spatiotemporal evidence across object pairs and refines relations using contextual priors derived from a world model (V-JEPA 2). Experiments on the 3DSSG and ReplicaSSG datasets demonstrate state-of-the-art (SoTA) performance in both object and predicate prediction, while producing temporally consistent scene structures. In particular, our method improves triplet recall by 77.4% and predicate recall by 23.2% over prior SoTA approaches, making it suitable for robotic manipulation and AR applications. Our code and models are open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。