剖析视觉语言模型计数失败原因并针对性提升性能
[De|Re]constructing VLMs' Reasoning in Counting
- 通过控制实验分析模型在计数任务中的表现与错误根源
- 仅微调输出层可使准确率提升最高21%
- 适用于关注模型推理机制与改进视觉理解的研究者
视觉语言模型（VLMs）在多个下游任务中表现出色，但其在视觉推理方面仍存在局限，如难以识别空间、时间关系及对象间关系，以及计数能力不足。本文通过受控实验，研究七种前沿VLMs在计数任务中的表现，发现其对物体数量、类型、空间排列及干扰项共现高度敏感。层间分析表明，错误主要源于最后一层表示到输出空间的错误映射。针对该问题，仅微调输出层即可使准确率提升最高21%。在真实世界数据集上也验证了该方法的一致有效性。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have recently gained attention due to their competitive performance on multiple downstream tasks, achieved by following user-input instructions. However, VLMs still exhibit several limitations in visual reasoning, such as difficulties in identifying relations (e.g., spatial, temporal, and among objects), understanding temporal sequences (e.g., frames), and counting objects. In this work, we go beyond score-level benchmark evaluations of VLMs by investigating the underlying causes of their failures and proposing a targeted approach to improve their reasoning capabilities. We study the reasoning skills of seven state-of-the-art VLMs in the counting task under controlled experimental conditions. Our experiments show that VLMs are highly sensitive to the number and type of objects, their spatial arrangement, and the co-occurrence of distractors. A layer-wise analysis reveals that errors are due to incorrect mapping of the last-layer representation into the output space. Our targeted training shows that fine-tuning just the output layer improves accuracy by up to 21%. We corroborate these findings by achieving consistent improvements on real-world datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。