实时生成3D语义场景图,速度远超现有方法。
FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images
- 直接将2D场景图升维到3D,用3D高斯分布表示物体
- 在ReplicaSSG和3DSSG上速度远超前人方法,性能更优
- 适合需要快速理解复杂环境的机器人、AR应用
将复杂的3D环境抽象为结构化表示对多个领域至关重要。3D语义场景图(SSGs)通过将物体作为节点、关系作为边来实现,支持高层场景理解。然而,现有3D SSG生成方法面临计算开销大、非增量处理等问题,难以适用于实时开放世界应用。为此,我们提出FROSS(Faster-than-Real-Time Online 3D Semantic Scene Graph Generation),一种在线且超实时的3D SSG生成新方法,其核心是直接将2D场景图映射至3D空间,并以3D高斯分布表示物体,避免依赖精确且高耗时的点云处理。此外,我们扩展了Replica数据集,添加物体间关系标注,构建了ReplicaSSG数据集用于全面评估FROSS。在ReplicaSSG与3DSSG数据集上的实验表明,FROSS在性能优越的同时,运行速度显著快于以往方法。代码与数据集已开源:https://github.com/Howardkhh/FROSS。
原文摘要 · Abstract (English)
The ability to abstract complex 3D environments into simplified and structured representations is crucial across various domains. 3D semantic scene graphs (SSGs) achieve this by representing objects as nodes and their interrelationships as edges, facilitating high-level scene understanding. Existing methods for 3D SSG generation, however, face significant challenges, including high computational demands and non-incremental processing that hinder their suitability for real-time open-world applications. To address this issue, we propose FROSS (Faster-than-Real-Time Online 3D Semantic Scene Graph Generation), an innovative approach for online and faster-than-real-time 3D SSG generation that leverages the direct lifting of 2D scene graphs to 3D space and represents objects as 3D Gaussian distributions. This framework eliminates the dependency on precise and computationally-intensive point cloud processing. Furthermore, we extend the Replica dataset with inter-object relationship annotations, creating the ReplicaSSG dataset for comprehensive evaluation of FROSS. The experimental results from evaluations on ReplicaSSG and 3DSSG datasets show that FROSS can achieve superior performance while operating significantly faster than prior 3D SSG generation methods. Our implementation and dataset are publicly available at https://github.com/Howardkhh/FROSS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。