无需训练，直接利用视觉语言模型生成开放世界场景图。
Open World Scene Graph Generation using Vision Language Models
- 将场景图生成视为零样本结构化推理问题，结合多模态提示与嵌入对齐。
- 在无训练数据下实现对未知物体和关系的识别，准确率超越传统方法。
- 适合需要快速适配新场景、无标注数据的研究者使用。
场景图生成（SGG）旨在识别图像中的物体并提炼其关键的成对关系。现有方法依赖特定数据集的监督学习，限制了其在开放世界场景中的应用，即面对新物体和/或新关系时表现不佳。即使采用大型视觉语言模型（VLMs）的方法，通常仍需针对基准数据集进行微调。本文提出开放世界场景图生成（Open-World SGG），一种无需训练、高效且模型无关的框架，直接利用预训练VLM的知识生成场景图，无需额外学习。将SGG视为零样本结构化推理问题，该方法结合多模态提示、嵌入对齐和轻量级配对优化策略，可在未见物体词汇表和关系集合上进行推理。为评估此设定，我们制定了开放世界评估协议，衡量在未见过的物体与关系下的性能。在Visual Genome、Open Images V6和全景场景图（PSG）数据集上的实验表明，预训练的VLM可在无需任务级训练的情况下完成关系理解。
原文摘要 · Abstract (English)
Scene-Graph Generation (SGG) seeks to recognize objects in an image and distill their salient pairwise relationships. Most methods depend on dataset-specific supervision to learn the variety of interactions, restricting their usefulness in open-world settings, involving novel objects and/or relations. Even methods that leverage large Vision Language Models (VLMs) typically require benchmark-specific fine-tuning. We introduce Open-World SGG, a training-free, efficient, model-agnostic framework that taps directly into the pretrained knowledge of VLMs to produce scene graphs with zero additional learning. Casting SGG as a zero-shot structured-reasoning problem, our method combines multimodal prompting, embedding alignment, and a lightweight pair-refinement strategy, enabling inference over unseen object vocabularies and relation sets. To assess this setting, we formalize an Open-World evaluation protocol that measures performance when no SGG-specific data have been observed either in terms of objects and relations. Experiments on Visual Genome, Open Images V6, and the Panoptic Scene Graph (PSG) dataset demonstrate the capacity of pretrained VLMs to perform relational understanding without task-level training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。