提升开放词汇场景图生成中的空间关系理解能力
LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
- 基于多模态大模型构建空间关系增强的指令数据集
- 在开放词汇场景图生成上召回率提升8.6%，均值召回提升28.4%
- 适合需要精确空间理解的视觉推理与通用图像分析任务
场景图生成（SGG）将视觉场景转换为结构化图表示，为复杂视觉任务提供深层理解。然而，现有SGG模型常忽略关键空间关系，且在开放词汇场景下泛化能力不足。为此，我们提出LLaVA-SpaceSGG，一种面向开放词汇SGG的多模态大语言模型，强化空间关系建模。我们构建了名为SpaceSGG的指令微调数据集，通过整合公开数据集并利用开源模型在数据构建流程中合成数据，融合物体位置、关系与深度信息，形成三种格式：空间场景图描述、问答对和对话。为增强多模态大模型能力向SGG任务的迁移，引入两阶段训练范式。实验表明，相比基线模型，LLaVA-SpaceSGG在召回率上提升8.6%，平均召回率提升28.4%。代码、数据集及训练模型已公开于GitHub：https://github.com/Endlinc/LLaVA-SpaceSGG。
原文摘要 · Abstract (English)
Scene Graph Generation (SGG) converts visual scenes into structured graph representations, providing deeper scene understanding for complex vision tasks. However, existing SGG models often overlook essential spatial relationships and struggle with generalization in open-vocabulary contexts. To address these limitations, we propose LLaVA-SpaceSGG, a multimodal large language model (MLLM) designed for open-vocabulary SGG with enhanced spatial relation modeling. To train it, we collect the SGG instruction-tuning dataset, named SpaceSGG. This dataset is constructed by combining publicly available datasets and synthesizing data using open-source models within our data construction pipeline. It combines object locations, object relations, and depth information, resulting in three data formats: spatial SGG description, question-answering, and conversation. To enhance the transfer of MLLMs' inherent capabilities to the SGG task, we introduce a two-stage training paradigm. Experiments show that LLaVA-SpaceSGG outperforms other open-vocabulary SGG methods, boosting recall by 8.6% and mean recall by 28.4% compared to the baseline. Our codebase, dataset, and trained models are publicly accessible on GitHub at the following URL: https://github.com/Endlinc/LLaVA-SpaceSGG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。