无需重建3D场景,用检索增强生成实现更精准的语义关系推理。
SGR3 Model: Scene Graph Retrieval-Reasoning Model in 3D

- 利用多模态大模型与检索增强生成,跳过传统3D重建流程。
- 在无训练基线中表现相当,媲美基于图神经网络的专家模型。
- 适合需要快速部署、依赖外部知识的机器人场景理解任务。
3D场景图提供了对象实体及其关系的结构化表示,有助于机器人进行高层语义理解,同时对人类也直观易懂。现有3D场景图生成方法通常结合场景重建与图神经网络(GNN),但需多模态数据且依赖启发式构建,限制了关系三元组预测。本文提出一种无需训练的3D场景图检索-推理模型(SGR3 Model),利用多模态大语言模型(MLLM)与检索增强生成(RAG),通过类似ColPali的跨模态框架检索语义对齐的场景图。该模型避免显式3D重建,通过加权块级相似性选择机制提升检索鲁棒性,抑制模糊或语义贫乏区域的影响。实验表明,SGR3 Model在训练自由基线中表现优异,达到与基于GNN的专家模型相当水平。消融研究显示,外部检索信息被显式融入词元生成过程,而非通过抽象隐式内化。
原文摘要 · Abstract (English)
3D scene graphs provide a structured representation of object entities and their relationships, enabling high-level interpretation and reasoning for robots while remaining intuitively understandable to humans. Existing approaches for 3D scene graph generation typically combine scene reconstruction with graph neural networks (GNNs). However, such pipelines require multi-modal data that may not always be available, and their reliance on heuristic graph construction can constrain the prediction of relationship triplets. In this work, we introduce a Scene Graph Retrieval-Reasoning Model in 3D (SGR3 Model), a training-free framework that leverages multi-modal large language models (MLLMs) with retrieval-augmented generation (RAG) for semantic scene graph generation. SGR3 Model bypasses the need for explicit 3D reconstruction. Instead, it enhances relational reasoning by incorporating semantically aligned scene graphs retrieved via a ColPali-style cross-modal framework. To improve retrieval robustness, we further introduce a weighted patch-level similarity selection mechanism that mitigates the negative impact of blurry or semantically uninformative regions. Experiments demonstrate that SGR3 Model achieves competitive performance compared to training-free baselines and on par with GNN-based expert models. Moreover, an ablation study on the retrieval module and knowledge base scale reveals that retrieved external information is explicitly integrated into the token generation process, rather than being implicitly internalized through abstraction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。