用大模型生成零样本场景图，突破传统数据偏见限制
PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Tasks
- 基于视觉语言模型和大语言模型，自动生成细粒度谓词
- 零样本下性能媲美最先进弱监督模型，部分任务超越有监督方法
- 模块化设计可适配任意数据集，适合开放词汇场景理解
在场景图生成（SGG）中，结构化表示通过物体节点和连接谓词从视觉输入中提取，支持多样下游任务的图像推理。尽管全监督SGG持续进步，但受限于有限标注数据和长尾谓词分布，存在训练偏差，导致谓词多样性差、下游性能下降。本文提出PRISM-0，一种零样本开放词汇的SGG框架，采用自底向上流程，利用基础模型捕捉广泛谓词。检测到的物体对经视觉语言模型描述，由大语言模型生成细粒度与粗粒度谓词，并通过视觉问答模型验证。该模块化、数据无关的设计可丰富如Visual Genome等现有数据集，生成多样化且无偏倚的场景图。在完全零样本设置下，PRISM-0在SGG基准上表现与最先进弱监督模型相当，甚至在句子到图检索等任务中超越最先进有监督方法。
原文摘要 · Abstract (English)
In Scene Graph Generation (SGG), structured representations are extracted from visual inputs as object nodes and connecting predicates, enabling image-based reasoning for diverse downstream tasks. While fully supervised SGG has improved steadily, it suffers from training bias due to limited curated data and long-tail predicate distributions, leading to poor predicate diversity and degraded downstream performance. We present PRISM-0, a zero-shot open-vocabulary SGG framework that leverages foundation models in a bottom-up pipeline to capture a broad spectrum of predicates. Detected object pairs are filtered, described via a Vision-Language Model (VLM), and processed by a Large Language Model (LLM) to generate fine- and coarse-grained predicates, which are then validated by a Visual Question Answering (VQA) model. PRISM-0 modular, dataset-independent design enriches existing SGG datasets such as Visual Genome and produces diverse, unbiased graphs. While operating entirely in a zero-shot setting, PRISM-0 achieves performance on par with state-of-the-art weakly-supervised models on SGG benchmarks and even state-of-the-art supervised methods in tasks such as Sentence-to-Graph Retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。