研究视觉语言模型如何根据场景上下文生成物体描述。
Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication
- 构建COOCo数据集，测试模型在不同场景与物体匹配度下的表现。
- 模型注意力随语义契合度呈非单调变化，中等契合时注意力反而下降。
- 高/低语义契合度下模型更依赖局部特征，适合研究视觉推理机制。
为探究视觉语言模型（VLM）在生成物体指代时对场景上下文的依赖程度及条件，我们引入了《常见物体脱离上下文》（COOCo）数据集，并在多个VLM上开展实验，考察不同场景-物体一致性水平和噪声干扰下的表现。结果表明，模型会根据场景-物体语义相关性和噪声水平自适应地利用场景上下文。基于跨模型的一致趋势，进一步分析模型注意力模式随目标-场景语义契合度的变化，发现准确分类与中层注意力增强至目标物体相关。同时观察到非单调依赖关系：中等契合度时注意力下降，而低或高契合度时则上升。这表明VLM在参考生成中动态平衡局部与上下文信息。数据集与代码已公开于https://github.com/cs-nlp-uu/scenereg。
原文摘要 · Abstract (English)
To what degree and under what conditions do VLMs rely on scene context when generating references to objects? To address this question, we introduce the $\textit{Common Objects Out-of-Context (COOCo)}$ dataset and conduct experiments on several VLMs under different degrees of scene-object congruency and noise. We find that models leverage scene context adaptively, depending on scene-object semantic relatedness and noise level. Based on these consistent trends across models, we turn to the question of how VLM attention patterns change as a function of target-scene semantic fit, and to what degree these patterns are predictive of categorisation accuracy. We find that successful object categorisation is associated with increased mid-layer attention to the target. We also find a non-monotonic dependency on semantic fit, with attention dropping at moderate fit and increasing for both low and high fit. These results suggest that VLMs dynamically balance local and contextual information for reference generation. Dataset and code are available here: $\href{https://github.com/cs-nlp-uu/scenereg}{https://github.com/cs-nlp-uu/scenereg}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。