让大模型先指物再数数，显著提升视觉计数能力
Getting to the Point: Pointing Improves LVLMs at Counting
- 先生成目标物体坐标，再据此推理计数
- 指物准确率超94%，计数正确率最高
- 适合需要精准空间感知的视觉任务
基于指物的方法将复杂任务分解为逐次定位与推理步骤。给定查询后，模型首先通过生成目标物体的坐标进行定位，再基于这些坐标预测答案。尽管该方法已被证明可提升大型视觉语言模型（LVLMs）性能，但其提升原因仍不明确。本文在零样本视觉场景计数任务中评估了多种微调与无训练方法，并与点-计数（PtC）方法对比，即模型先生成目标物体的坐标，再预测数量。结果表明，PtC在所有方法中表现最佳，预测坐标在超过94%的情况下正确对齐图像（以F1-score衡量）。机制分析显示，性能提升源于坐标所携带的空间信息。然而，定位性能在图像不同区域间存在差异，揭示出空间偏差。最终结果表明，PtC在合成与真实数据上均提升了分布外泛化能力，表明坐标有助于增强LVLM的计数能力。
原文摘要 · Abstract (English)
Pointing-based methods decompose complex tasks as sequential grounding and reasoning steps. Given a query, the model first grounds the relevant objects by generating their coordinates, and then predicts an answer conditioned on these points. While this approach has been shown to increase the performance of Large Vision-Language Models (LVLMs), it remains unclear why and how it improves the models' visual reasoning. In this work, we evaluate pointing-based methods in the task of zero-shot counting in visual scenes. We experiment with multiple fine-tuning and training-free approaches on state-of-the-art LVLMs, and compare them with Point-then-Count (PtC), where models first generate point coordinates for the target objects and then predict their count. Our results show that PtC achieves the highest accuracy among the evaluated approaches, with predicted points correctly grounded in the image in more than 94% of cases (based on F1-score). Mechanistic analyses show that gains arise from spatial information encoded in the predicted coordinates. Nevertheless, grounding performance varies across image regions, revealing spatial biases. Finally, the results indicate that PtC improves out-of-distribution generalization on both synthetic and real data, suggesting the potential of coordinates to help LVLMs improve their counting skills.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。