arXiv:2604.12119cs.CVcs.LG2026-04

发现大模型会固守默认语义解释,即使提示有其他合理答案。

Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

  • 设计抽象棋类游戏基准,区分感知错误与语义固定问题。
  • 14个模型在反向规则下准确率平均下降23%,显示显著语义固化现象。
  • 改写提示词可缓解问题,适合研究模型推理机制的学者参考。

大型视觉语言模型常依赖熟悉语义先验,但现有评估无法清晰区分感知错误与规则映射错误。本文将此行为称为语义固定:即使提示给出另一种同样合理的映射,模型仍坚持默认解释。为隔离该效应,提出VLM-Fix基准,基于四种抽象策略游戏,在相同终局状态下对比标准规则与逆向规则的性能。在14个开放与闭源模型中,标准规则下的准确率始终更高,揭示稳定的语义固定差距。提示干预验证该机制:中性别名提示显著缩小逆向规则差距,而语义负载别名则使其重现。后训练表明:单一规则训练提升同规则迁移,损害反规则迁移;联合规则训练则促进更广迁移。为检验外部有效性,对VLMBias进行类似去熟悉化干预,观察到相同定性模式。晚期激活引导部分恢复性能下降,表明语义固定错误至少部分存在于后期表征中。项目页面、代码与数据集详见https://maveryn.github.io/vlm-fix/。

原文摘要 · Abstract (English)

Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures. We study this behavior as semantic fixation: preserving a default interpretation even when the prompt specifies an alternative, equally valid mapping. To isolate this effect, we introduce VLM-Fix, a controlled benchmark over four abstract strategy games that evaluates identical terminal board states under paired standard and inverse rule formulations. Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap. Prompt interventions support this mechanism: neutral alias prompts substantially narrow the inverse-rule gap, while semantically loaded aliases reopen it. Post-training is strongly rule-aligned: training on one rule improves same-rule transfer but hurts opposite-rule transfer, while joint-rule training improves broader transfer. To test external validity beyond synthetic games, we evaluate analogous defamiliarization interventions on VLMBias and observe the same qualitative pattern. Finally, late-layer activation steering partially recovers degraded performance, indicating that semantic-fixation errors are at least partly editable in late representations. Project page, code, and dataset available at https://maveryn.github.io/vlm-fix/.

视觉语言模型语义固定模型偏差

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。