arXiv:2412.05596cs.CV2024-12AAAI被引 6

构建可变功能的3D场景图,让机器理解空间与物体的用途关系。

TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances

  • 用分层图结构表示房间、区域和物体的空间关系与功能
  • 在自建数据集上实现比基线模型更高的场景理解准确率
  • 适合研究3D场景理解与具身智能的开发者和研究人员

功能与可用性是3D场景理解的关键,支持任务导向目标。本文提出一种模型,学习在表示场景空间结构的3D分层场景图(3DHSG)中动态建模功能可用性。该模型将变化的功能可用性与图中不同的空间上下文融合。具体而言,从分割后的点云和语义标签出发,构建包含顶层房间节点、子节点定义室内局部区域及其区域特定功能、孙节点标识物体位置及其物体特定功能的3DHSG。为支持研究,我们构建了一个定制的3DHSG数据集,提供局部区域的功能标注和物体的功能标注。采用基于Transformer的模型学习3DHSG,并使用多任务学习框架同时完成房间分类和区域内功能划分。实验表明,该方法优于现有基线模型,展示了将Transformer应用于3D场景理解与3DHSG生成的有效路径。代码与数据集已公开。

原文摘要 · Abstract (English)

The concept of function and affordance is a critical aspect of 3D scene understanding and supports task-oriented objectives. In this work, we develop a model that learns to structure and vary functional affordance across a 3D hierarchical scene graph representing the spatial organization of a scene. The varying functional affordance is designed to integrate with the varying spatial context of the graph. More specifically, we develop an algorithm that learns to construct a 3D hierarchical scene graph (3DHSG) that captures the spatial organization of the scene. Starting from segmented object point clouds and object semantic labels, we develop a 3DHSG with a top node that identifies the room label, child nodes that define local spatial regions inside the room with region-specific affordances, and grand-child nodes indicating object locations and object-specific affordances. To support this work, we create a custom 3DHSG dataset that provides ground truth data for local spatial regions with region-specific affordances and also object-specific affordances for each object. We employ a transformer-based model to learn the 3DHSG. We use a multi-task learning framework that learns both room classification and learns to define spatial regions within the room with region-specific affordances. Our work improves on the performance of state-of-the-art baseline models and shows one approach for applying transformer models to 3D scene understanding and the generation of 3DHSGs that capture the spatial organization of a room. The code and dataset are publicly available.

3D场景理解分层图功能可用性Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。