用大模型生成关系假设,再通过视觉证据迭代修正,让模型学会识别未标注的新互动。
Hallucinate, Ground, Repeat: A Framework for Generalized Visual Relationship Detection
- 利用大模型生成潜在关系,再用视觉信息验证并优化,形成循环迭代的推理机制。
- 在21个未标注关系上达到15.9的平均召回率,显著优于现有方法。
- 适合需要理解开放世界复杂互动的场景,如机器人导航或智能助手。
理解物体间的关系是视觉智能的核心,广泛应用于具身智能、辅助系统和场景理解。然而,大多数视觉关系检测(VRD)模型依赖固定谓词集,难以泛化到新交互。关键挑战在于无法对基于外部知识推测出的语义合理但未标注的关系进行视觉定位。本文提出一种基于大语言模型(LLM)的迭代视觉定位框架,灵感来自期望最大化(EM)算法:先用LLM从检测到的物体生成候选场景图(期望阶段),再训练视觉模型将这些假设与感知证据对齐(最大化阶段)。该过程可逐步扩展关系理解范围,实现对未见谓词的泛化。此外,我们在视觉基因组数据集上构建了新的开放世界VRD基准,包含21个保留的谓词,并在已见、未见和混合三种设置下评估。所提模型在三种设置下的谓词分类平均召回率(mR@50)分别为15.9、13.1和11.7,优于仅用LLM、少样本及去偏基线方法,证明了基于视觉约束的LLM先验在可扩展开放世界视觉理解中的潜力。
原文摘要 · Abstract (English)
Understanding relationships between objects is central to visual intelligence, with applications in embodied AI, assistive systems, and scene understanding. Yet, most visual relationship detection (VRD) models rely on a fixed predicate set, limiting their generalization to novel interactions. A key challenge is the inability to visually ground semantically plausible, but unannotated, relationships hypothesized from external knowledge. This work introduces an iterative visual grounding framework that leverages large language models (LLMs) as structured relational priors. Inspired by expectation-maximization (EM), our method alternates between generating candidate scene graphs from detected objects using an LLM (expectation) and training a visual model to align these hypotheses with perceptual evidence (maximization). This process bootstraps relational understanding beyond annotated data and enables generalization to unseen predicates. Additionally, we introduce a new benchmark for open-world VRD on Visual Genome with 21 held-out predicates and evaluate under three settings: seen, unseen, and mixed. Our model outperforms LLM-only, few-shot, and debiased baselines, achieving mean recall (mR@50) of 15.9, 13.1, and 11.7 on predicate classification on these three sets. These results highlight the promise of grounded LLM priors for scalable open-world visual understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。