VLM模型数数能力差，因视觉信息在语言阶段被弱化。
Counting to Four is still a Chore for VLMs
- 设计可控计数测试集，暴露模型在不同图像布局下的弱点。
- 发现视觉证据在语言层显著减弱，文本先验干扰严重。
- 提出轻量干预方法MAS，提升视觉注意力，改善计数表现。
视觉-语言模型（VLM）在复杂多模态推理任务中表现优异，但在物体计数等基础定位技能上仍存在失败。现有评估主要关注最终输出，难以揭示模型内部错误根源。本文通过行为与机制分析，构建了名为COUNTINGTRICKS的受控评估套件，包含基于简单形状的计数案例，用以暴露模型在不同图像分块布局和对抗性提示下的脆弱性。通过注意力分析与组件探针，我们发现与计数相关的视觉证据在模态投影阶段最强，但在后续语言层显著退化，模型更易受文本先验影响。基于此，我们进一步评估了轻量级干预方法「模态注意力共享」（MAS），该方法在生成答案时强制保留最小视觉注意力预算。结果表明，计数失败不仅源于视觉感知局限，更在于语言阶段对视觉证据的利用不足。代码与数据集将公开于https://github.com/leduy99/-CVPRW26-Modality-Attention-Share。
原文摘要 · Abstract (English)
Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。