利用视觉语言模型注意力结构，无需微调即可提升小物体定位准确率
Self-Improving Small Object Grounding in LVLMs
- 基于注意力图训练轻量级回归器预测框匹配度
- 在COCO和Objects365上实现最高19%的小物体定位提升
- 提出无训练候选框选择器，兼具高效与可解释性
大型视觉语言模型（LVLMs）内部的注意力模式能否在不进行微调的情况下识别出可靠的小物体边界框？本文给出肯定回答。LVLM中的注意力结构编码了定位质量——仅基于注意力图训练的轻量级交并比（IoU）回归器即可实现强预测性能（皮尔逊相关系数 r > 0.67）。该回归器驱动我们提出的基于回归的注意力候选选择框架（ACS-Learned），从多个采样候选框中选出最优框以改善物体定位。通过分析回归器学习内容，我们发现特定的Transformer层和注意力头最为关键，并据此提出无需训练的选框方法（ACS-Free）：仅通过这些关键头的注意力熵对候选框排序，推理时无任何可学习组件。在COCO和Objects365上的实验表明，该方法可实现最高19%的自提升性能，且在所有无训练方法中排名第一，证明了注意力结构不仅能提升定位可靠性，还增强了模型可解释性。
原文摘要 · Abstract (English)
Can internal attention patterns in Large Vision Language Models (LVLMs) identify reliable small-object boxes without fine-tuning? In this work, we provide an affirmative answer. Attention structure in LVLMs encodes grounding quality-a lightweight IoU regressor trained solely on attention maps achieves strong IoU prediction (Pearson r > 0.67). This regressor powers the regressor-based variant of our Attention-based Candidate Selection (ACS) framework, called ACS-Learned, which selects the best box from multiple sampled candidates to improve object grounding. By analyzing what the regressor learns, we reveal which transformer layers and heads are most critical and derive ACS-Free: a training-free selector that ranks candidates by attention entropy on these discriminative heads, with no learned component at inference. Experiments on COCO and Objects365 demonstrate up to 19% self-improvement on small object localization, with ACS-Free ranking best among all training-free methods, demonstrating that useful attention structure improves both localization reliability and interpretability in LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。