用符号锚定测试视觉语言模型对成语义的理解能力
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
- 构建DIVA基准，用图标化图像对比字面与成语义的视觉表征
- 发现模型普遍存在字面优先偏差，视觉越真实反而越难理解成语
- 适合研究模型语义理解、视觉抽象与认知偏差的学者参考
视觉语言模型在生成逼真图像方面表现优异，但在表达抽象含义（如名词复合词的习语意义）时仍存在困难。为探究高视觉保真度是否干扰视觉抽象下的习语组合性理解，我们提出DIVA基准：通过生成字面义与习语义对应的、具有语义锚定的图标化图像，实现可控对比。进一步提出语义对齐差距（Δ），一种与架构无关的指标，用于量化字面与习语视觉定位之间的偏离程度；并引入方向性符号偏置b(t)，分别测量字面偏好方向与强度。评估8个近期VLMs后发现，模型规模无法消除字面优先偏差，且视觉保真度越高，符号对齐越弱，表明超现实图像可能造成认知干扰。结果表明，提升组合理解需对视觉输入进行图标化抽象，并将解释与生成锚定于预期意义。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel at photorealistic generation, yet often struggle to represent abstract meaning such as idiomatic interpretations of noun compounds. To study whether high visual fidelity interferes with idiomatic compositionality under visual abstraction, we introduce DIVA, a controlled benchmark that replaces high-fidelity visual detail with schematic iconicity by generating paired, sense-anchored visualizations for literal and idiomatic readings. We further propose Semantic Alignment Gap ($Δ$), an architecture-agnostic metric that quantifies divergence between literal and idiomatic visual grounding. We additionally introduce a directional signed bias $b(t)$ to separately measure the direction and strength of literal preference. Evaluating 8 recent VLMs, we reveal a consistent Literal Superiority Bias: model scale alone does not resolve literal preference, and increased visual fidelity is associated with weaker symbolic alignment, suggesting cognitive interference from hyper-realistic imagery. Our findings suggest that improving compositional understanding requires iconographic abstraction of visual input and anchoring interpretation and generation in intended meaning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。