测试视觉语言模型计数能力，发现复杂度越高越容易出错。
Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions
- 构建合成数据集，系统测试模型在不同图像和提示下的计数表现。
- 计数准确率随物体数量、颜色、纹理等复杂度上升而下降。
- 调整语言模型注意力可轻微提升计数效果，适合研究模型认知机制。
近期研究表明，视觉语言模型（VLMs）在回答图像视觉属性问题时常依赖训练中习得的固有偏见。当模型被问及需要选择性视觉注意的高精度问题时，这种偏见会被放大，类似人类计数任务中的认知挑战。本文构建了一个合成基准数据集与评估框架，系统分析计数性能如何随图像和提示属性变化。使用开源VLMs，我们考察了物体数量、颜色、背景色、物体纹理、背景纹理及提示具体性等可控扰动下的性能变化，并分析对应的视觉注意力分配。进一步在语言模型解码器中进行探索性注意力重加权实验，调节不同层对视觉标记的关注度，评估其对计数行为的影响。结果表明，计数准确率随视觉与语言复杂度增加而系统性下降，呼应人类认知极限与负荷效应；而针对性注意力重加权带来适度但可观测的改进。本研究不追求基准准确率竞赛，而是引入一种受控诊断框架，用于分析VLM的计数行为。通过系统实验，揭示了源于跨模态绑定的失败模式，这些模式在自然图像基准中难以察觉，并提供初步实证证据：在语言解码器中实施注意力重加权，可影响模型将语言量词概念锚定于视觉表征的方式。代码与数据见：https://github.com/ssen7/vlm-count-analysis
原文摘要 · Abstract (English)
Recent research suggests that Vision Language Models (VLMs) often rely on inherent biases learned during training when responding to queries about visual properties of images. These biases are exacerbated when VLMs are asked highly specific questions that require selective visual attention, a demand that mirrors cognitive challenges observed in human enumeration tasks. We build upon this research by developing a synthetic benchmark dataset and evaluation framework to systematically characterize how counting performance varies as image and prompt properties change. Using open-source VLMs, we analyze how performance shifts across controlled perturbations (e.g. number of objects, object color, background color, object texture, background texture, and prompt specificity) and examine corresponding changes in visual attention allocation. We further conduct exploratory attention reweighting experiments in the language model decoder to modulate focus on visual tokens at different layers and assess their effects on counting behavior. Our results reveal that counting accuracy degrades systematically with increasing visual and linguistic complexity echoing human limits and cognitive load effects known from human perception, while targeted attention reweighting yields modest but measurable improvements. Rather than competing on benchmark accuracy, we introduce a controlled diagnostic framework for analyzing VLM enumeration behavior. Through systematic experiments, we expose failure modes rooted in cross-modal binding that natural image benchmarks may not easily isolate, and provide preliminary empirical evidence that targeted attention reweighting in the language decoder can influence how models ground linguistic quantity concepts in visual representations. Code and data available here: https://github.com/ssen7/vlm-count-analysis
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。