arXiv:2604.09687cs.CVcs.AI2026-04

发现视觉语言模型在识别彩色网格时存在细节丢失问题,称为'数字失认'。

Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models

  • 设计网格转矩阵任务,精确测试模型对视觉细节的捕捉能力。
  • 小尺寸网格即出现严重错误,而非渐进退化,显示模型脆性。
  • 错误具有结构规律,与图像块边界重叠有关,适合界面等场景评估。

视觉语言模型(VLMs)在多模态推理基准上表现优异,但这些评估通常不要求对图像进行全量读取,因而掩盖了其在忠实捕捉所有视觉细节方面的失败。我们提出Grid2Matrix(G2M),一个受控基准:模型需根据彩色网格和颜色到数字的映射,输出对应矩阵。通过调整网格大小和颜色数量,G2M在最小化语义混淆的前提下提升视觉复杂度。实验发现,VLMs在零样本端到端评估中出现急剧的早期崩溃,甚至在非常小的网格上就失败,而非随任务密度增加而渐进退化。我们对两类代表性VLM的视觉编码器进行探查,发现其保留的网格信息远多于最终的语言输出。这表明失败原因不仅在于视觉编码,更源于视觉特征可恢复性与语言表达之间存在差距,我们称之为‘数字失认’。进一步分析显示,这些错误高度结构化,强烈依赖网格单元与视觉块边界的位置关系。此外,模型缩放和多模态对齐等常见策略无法完全消除此缺陷。我们期望G2M能成为理解VLM如何丢失精细视觉细节的有用测试平台,并用于评估表格、图表、表单和图形用户界面等对微小视觉细节敏感的任务。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap \textit{Digital Agnosia}. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.

视觉语言模型细节丢失评估基准数字失认

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。