用概率图推理构建3D场景功能关系,让机器像人一样理解物体间整体关联。
FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning
- 将物体和部件的3D地图与基础模型生成的功能关系构建成因子图
- 在多个数据集上提升关系发现召回率,显著降低模糊关系的置信度误差
- 适合做3D场景理解、机器人交互等需要常识推理的任务
当前3D场景理解正从纯空间分析转向功能性理解。然而现有方法常孤立地分析物体对之间的功能关系,无法捕捉人类用于消歧的全局依赖性。我们提出FunFact,一个从带姿态的RGB-D图像构建概率性开放词汇功能3D场景图的框架。FunFact首先建立以物体和部件为中心的3D地图,并利用基础模型生成语义合理的功能关系候选。这些候选被转化为因子图变量,并受大语言模型提供的常识先验与几何先验约束。该建模方式支持所有功能边及其边缘概率的联合推断,显著提升了置信度校准效果。为评估此设定,我们引入FunThor,一个基于AI2-THOR的合成数据集,包含部件级几何结构和基于规则的功能标注。在SceneFun3D、FunGraph3D和FunThor上的实验表明,FunFact提高了节点与关系发现的召回率,显著降低了模糊关系的校准误差,凸显了整体概率建模在功能性场景理解中的优势。
原文摘要 · Abstract (English)
Recent work in 3D scene understanding is moving beyond purely spatial analysis toward functional scene understanding. However, existing methods often consider functional relationships between object pairs in isolation, failing to capture the scene-wide interdependence that humans use to resolve ambiguity. We introduce FunFact, a framework for constructing probabilistic open-vocabulary functional 3D scene graphs from posed RGB-D images. FunFact first builds an object- and part-centric 3D map and uses foundation models to propose semantically plausible functional relations. These candidates are converted into factor graph variables and constrained by both LLM-derived common-sense priors and geometric priors. This formulation enables joint probabilistic inference over all functional edges and their marginals, yielding substantially better calibrated confidence scores. To benchmark this setting, we introduce FunThor, a synthetic dataset based on AI2-THOR with part-level geometry and rule-based functional annotations. Experiments on SceneFun3D, FunGraph3D, and FunThor show that FunFact improves node and relation discovery recall and significantly reduces calibration error for ambiguous relations, highlighting the benefits of holistic probabilistic modeling for functional scene understanding. See our project page at https://funfact-scenegraph.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。