arXiv:2603.23118cs.CVcs.MM2026-03被引 1

提出多尺度感知策略,让多模态大模型看清人类能发现的视觉幻觉。

SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions

  • 通过抑制高频背景干扰,模拟人类多尺度视觉感知。
  • 使Qwen3-VL-8B-Instruct在幻觉图像上准确率从13.0%提升至84.0%。
  • 适合关注模型安全与视觉感知对齐的研究者使用。

近期研究发现,多模态大语言模型(MLLMs)极易受隐藏模式视觉幻觉影响,此类幻觉中隐藏内容对人类明显却难以被模型察觉,暴露出模型与人类在感知上的错位及潜在安全风险。为系统研究该问题,我们构建了综合性强、挑战性高的幻觉数据集IlluChar,揭示了模型失败的关键机制:高频注意力偏移,即模型易被幻觉图像中的高频背景纹理干扰,从而忽略隐藏图案。为此,我们提出一种即插即用的多尺度感知策略(SMSP),其设计契合人类视觉感知规律。通过抑制干扰性高频背景信号,生成更接近人类感知的图像表示。实验表明,SMSP显著提升了所有评测模型在幻觉图像上的表现,例如将Qwen3-VL-8B-Instruct的准确率从13.0%提升至84.0%。本工作为理解MLLM视觉感知提供了新视角,并提供了一种实用且鲁棒的增强方案。代码已开源:https://github.com/Tujz2023/SMSP。

原文摘要 · Abstract (English)

Recent works have shown that multimodal large language models (MLLMs) are highly vulnerable to hidden-pattern visual illusions, where the hidden content is imperceptible to models but obvious to humans. This deficiency highlights a perceptual misalignment between current MLLMs and humans, and also introduces potential safety concerns. To systematically investigate this failure, we introduce IlluChar, a comprehensive and challenging illusion dataset, and uncover a key underlying mechanism for the models' failure: high-frequency attention bias, where the models are easily distracted by high-frequency background textures in illusion images, causing them to overlook hidden patterns. To address the issue, we propose the Strategy of Multi-Scale Perception (SMSP), a plug-and-play framework that aligns with human visual perceptual strategies. By suppressing distracting high-frequency background signals, SMSP generates images closer to human perception. Our experiments demonstrate that SMSP significantly improves the performance of all evaluated MLLMs on illusion images, for instance, increasing the accuracy of Qwen3-VL-8B-Instruct from 13.0% to 84.0%. Our work provides novel insights into MLLMs' visual perception, and offers a practical and robust solution to enhance it. Our code is publicly available at https://github.com/Tujz2023/SMSP.

多模态视觉幻觉感知对齐即插即用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。