通过反向对比注意力提升视觉语言模型的指代检测能力
Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention
- 用反向对比机制重加权注意力，强化语义相关但弱激活的特征
- 在15个模型中11个提升性能，最高增益达26.6%
- 无需重训练，适合需要可解释性的多模态应用
我们提出反向对比注意力（RCA），一种无需重新训练即可增强视觉语言模型目标定位能力的即插即用方法。RCA通过抑制极端激活并放大中等强度的注意力值，使语义相关但较弱的特征主导预测。我们在开放词汇指代检测（OV-RefOD）任务上进行评估，引入无置信度的平均精度指标FitAP，基于交并比和框面积计算。RCA在15个开源视觉语言模型中提升了11个，最高提升达+26.6%。性能提升与注意力锐度和融合时机相关：晚期融合模型普遍受益，而如DeepSeek-VL2等模型也实现改进，表明模型容量与特征解耦是关键因素。RCA同时提供可解释性与性能提升。代码与数据集已公开于https://github.com/earl-juanico/rca。
原文摘要 · Abstract (English)
We propose Reverse Contrast Attention (RCA), a plug-in method that enhances object localization in vision-language transformers without retraining. RCA reweights final-layer attention by suppressing extremes and amplifying mid-level activations to let semantically relevant but subdued tokens guide predictions. We evaluate it on Open Vocabulary Referring Object Detection (OV-RefOD), introducing FitAP, a confidence-free average precision metric based on IoU and box area. RCA improves FitAP in 11 out of 15 open-source VLMs, with gains up to $+26.6\%$. Effectiveness aligns with attention sharpness and fusion timing; while late-fusion models benefit consistently, models like $\texttt{DeepSeek-VL2}$ also improve, pointing to capacity and disentanglement as key factors. RCA offers both interpretability and performance gains for multimodal transformers. Codes and dataset are available from https://github.com/earl-juanico/rca
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。