用颜色错位测试视觉语言模型的感知弱点，发现模型易被干扰而犯错。
Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models
- 设计八种颜色混合干扰，模拟视觉错觉测试模型感知能力
- 模型准确率随干扰强度急剧下降，大模型也难避免
- 人类表现远超模型，简单预处理可部分恢复模型性能
视觉语言模型（VLMs）在基准测试中表现优异，但存在系统性感知缺陷：对像素值进行结构化、大幅度修改后，模型仍会给出自信却荒谬的预测，而人类对此类场景仍能轻松识别。本文通过空间色彩混合（Spatial Colour Mixing）这一程序化颜色失真方法，研究该差距。该方法在自然图像上叠加结构化图案（使用RGB与Ostwald色彩系统）。我们构建了八种空间色彩混合变体，评估了九个VLMs在三个模型家族上的表现，覆盖四个数据集。结果显示，随着干扰强度增加，所有模型的准确率均显著下降，扩大语言模型规模无法稳定缓解失败。在包含61名参与者的真人实验中，人类在动物识别任务上明显优于模型。此外，一种受人类启发的简单预处理方法，能有效恢复部分性能，提示感知友好型预处理和工具使用是提升VLM鲁棒性的实用策略。
原文摘要 · Abstract (English)
Vision-language models (VLMs) achieve strong benchmark results, yet can exhibit systematic perceptual weaknesses: structured, large changes to pixel values can cause confident yet nonsensical predictions, even when the underlying scene remains easily recognizable to humans. We study this gap using Spatial Colour Mixing, a programmatic family of colour distortions that overlays structured patterns (in both RGB and Ostwald colour systems) onto natural images. We introduce a framework of eight spatial colour mixing variants and evaluate nine VLMs across three model families on four datasets. Across models and datasets, accuracy degrades sharply with increasing distortion, and scaling the language model does not reliably mitigate the failure. In a human study with 61 participants on an animal recognition dataset, humans substantially outperform VLMs under the same distortions. Finally, we show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。