让视觉模型更准地判断物体空间关系,通过显式构建查询对应的证据。
SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification

- 在推理时构造仅针对当前问题的视觉证据,避免错误匹配。
- 在多个数据集上提升准确率,最高增益达11.79个百分点。
- 适合需要精准空间推理的应用,如智能问答和图像理解。
空间关系问题要求模型先识别查询中的主体和客体,再比较其布局。然而,即使视觉语言模型能识别两个实体,仍可能基于错误实例或模糊全局视图作答。我们探讨是否通过显式构建查询相关的证据来缓解这一缺陷,提出SEER(自接地实体-关系推理证据接口),一种无需训练、仅在推理阶段使用的接口,适用于冻结的视觉语言模型。SEER在成对定位时隐藏候选关系,构建包含明确主体/客体角色的查询特定视图,并保留完整图像与稀疏框几何作为互补证据。对于支持精确逆关系的判断协议,可选精炼步骤交换实体角色,仅当恰好一个视觉状态满足对应逆关系时才改变原决策。在图像不重叠的GQA-Train900测试集上,相比完整模型,SEER提升3.94 [2.17,5.72];该收益在标签无关的定位顺序平衡下依然为正,且在535个实体名称唯一的问题中保持有效。不变协议下,所有2,434个筛选后的EmbSpatial成对关系问题上,提升范围为+4.35至+11.79,涵盖三种模型。对照实验分离了局部聚焦与角色显式条件的作用。结果确立查询特定证据构建为关键干预,互逆一致性仅为次级协议优化。
原文摘要 · Abstract (English)
Spatial relation questions require a model to identify the queried subject and object before comparing their layout. Yet a VLM can recognize both entities and still answer from the wrong instance or an ambiguous global view. We ask whether making query-specific evidence explicit can mitigate this failure and propose SEER (Self-grounded Evidence for Entity-Relation Reasoning), a training-free inference-time evidence interface for frozen VLMs. SEER hides candidate relations during pair localization, constructs a query-specific view with explicit subject/object roles, and retains the full image and sparse box geometry as complementary evidence. For relation-choice protocols with exact inverse support, an optional refinement swaps the entity roles and changes the forward decision only when exactly one visual state obeys the corresponding inverse relation. On an image-disjoint GQA-Train900 test frozen before model scoring, SEER pools to +3.94 [2.17,5.72] over Full; the gain remains positive under label-independent grounding-order counterbalancing and on the 535 rows whose entity names are unique. The unchanged protocol yields +4.35 to +11.79 on all 2,434 filtered EmbSpatial pair-relation questions across three models. Matched controls separate local refocus from role-explicit conditioning. These results establish query-specific evidence construction as the principal intervention, with reciprocal consistency as a smaller protocol-specific refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。