arXiv:2510.12287cs.CVcs.CL2025-10被引 1

发现视觉语言模型会误认图标中的文字,因投影层存在符号先验。

Vision Language Models Map Logos to Text via Semantic Entanglement in the Visual Projector

  • 通过分析投影层维度,发现模型依赖符号先验而非真实字符感知
  • 在强干扰下仍出现幻觉,遮挡时错误率最高达72%
  • 适合关注多模态模型可信度与幻觉问题的研究者

视觉语言模型(VLMs)在多模态推理中取得显著进展,但仍易产生幻觉,即输出与视觉证据不符。本文研究了此前被忽视的场景:图标幻觉,即模型在无文字的图标中生成品牌名或文本内容。我们使用纯符号、混合及含文字图标的数据集,以及具有挑战性的Hard-60子集,系统评估了主流VLMs的幻觉现象。通过九种结构化扰动测试,发现即使在强扭曲下幻觉依然存在,遮挡条件下表现最差。对开源的LLaVA模型进行嵌入级分析表明,幻觉与投影器中少数维度密切相关,针对性删减可显著降低错误率,同时保持光学字符识别(OCR)准确性。结果揭示,VLMs常依赖符号先验而非真实字形感知,尤其在圆形标识上表现明显,且投影子空间在此失败模式中起决定性作用。本工作提供了新的诊断视角和可操作的缓解策略,强调投影解耦与基于OCR的解码是提升多模态系统可信度的可行方向。

原文摘要 · Abstract (English)

Vision Language Models (VLMs) have achieved impressive progress in multimodal reasoning; yet, they remain vulnerable to hallucinations, where outputs are not grounded in visual evidence. In this paper, we investigate a previously overlooked setting: logo hallucination, where models generate brand names or textual content despite logos containing no visible words. Using curated splits of pure symbols, hybrids, and text-bearing logos, as well as the challenging Hard-60 subset, we systematically measure hallucination across leading VLMs. We further probe robustness through nine structured perturbations and show that hallucinations persist even under strong distortions, with occlusion exposing the sharpest weaknesses. Embedding-level analysis with open-weight LLaVA demonstrates that hallucination is tied to a small subset of projector dimensions, and targeted ablation substantially reduces errors while preserving OCR accuracy. Together, these findings reveal that VLMs often rely on symbolic priors rather than genuine glyph perception, particularly for iconic circular logos, and that projector subspaces play a decisive role in this failure mode. Our work contributes both a novel diagnostic lens and actionable mitigation insights, highlighting projector disentanglement and OCR-guided decoding as promising directions for building more trustworthy multimodal systems.

多模态幻觉检测投影层图标识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。