用场景图桥接视觉与语言,实现零样本指代理解的可解释推理。
Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs
- 以查询驱动构建带空间关系的场景图作为中间表示。
- 在多个基准上达顶尖准确率,如RefCOCOg val达73.28%。
- 适合需要透明决策过程的高阶视觉语言任务使用者。
零样本指代理解旨在不依赖特定任务训练数据的情况下,根据自然语言查询定位图像中的目标物体,要求强大的视觉理解能力。现有视觉语言模型(如CLIP)通常通过直接计算文本查询与图像区域之间的特征相似性来解决此问题,但难以捕捉细粒度视觉细节和复杂对象关系。同时,大语言模型虽擅长高层次语义推理,却无法直接将视觉特征映射为文本语义,限制其在指代理解中的应用。为此,我们提出SGREC,一种基于查询驱动场景图的可解释零样本指代理解方法。首先利用视觉语言模型构建一个显式编码空间关系、描述性标题及对象交互的查询驱动场景图;再借助该场景图,弥合低层图像区域与高层语义理解之间的差距;最后由大语言模型基于场景图提供的结构化文本表示推断目标物体,并给出详细决策解释,确保推理过程可解释。大量实验表明,SGREC在多数零样本指代理解基准上达到领先性能,包括RefCOCO val(66.78%)、RefCOCO+ testB(53.43%)和RefCOCOg val(73.28%),凸显其出色的视觉场景理解能力。
原文摘要 · Abstract (English)
Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries without relying on task-specific training data, demanding strong visual understanding capabilities. Existing Vision-Language Models~(VLMs), such as CLIP, commonly address zero-shot REC by directly measuring feature similarities between textual queries and image regions. However, these methods struggle to capture fine-grained visual details and understand complex object relationships. Meanwhile, Large Language Models~(LLMs) excel at high-level semantic reasoning, their inability to directly abstract visual features into textual semantics limits their application in REC tasks. To overcome these limitations, we propose \textbf{SGREC}, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries. Specifically, we first employ a VLM to construct a query-driven scene graph that explicitly encodes spatial relationships, descriptive captions, and object interactions relevant to the given query. By leveraging this scene graph, we bridge the gap between low-level image regions and higher-level semantic understanding required by LLMs. Finally, an LLM infers the target object from the structured textual representation provided by the scene graph, responding with detailed explanations for its decisions that ensure interpretability in the inference process. Extensive experiments show that SGREC achieves top-1 accuracy on most zero-shot REC benchmarks, including RefCOCO val (66.78\%), RefCOCO+ testB (53.43\%), and RefCOCOg val (73.28\%), highlighting its strong visual scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。