用稀疏注意力提升CLIP的细粒度图像理解能力
Sparse Attention for Dense Open-Vocabulary Prediction in CLIP
- 用α-entmax替换CLIP最后层的softmax，实现注意力稀疏化
- 在多个开放词汇任务上显著提升分割与检索性能
- 特别适合需要精准空间定位的视觉理解场景
对比语言-图像预训练（CLIP）依赖基于softmax的自注意力机制，该机制对所有词元对分配概率质量，即使语义无关也如此。虽然这种密集分布有助于预训练时获取全局上下文，但会将注意力分散到大量低显著性词元上，产生噪声，掩盖细粒度、空间局部化的线索，影响密集开放词汇预测。本文在推理阶段将最终视觉自注意力层中的行级softmax替换为α-entmax变换，应用于标准查询-键注意力和自相关变体。由于entmax采用数据依赖阈值，将低分项精确置零，起到隐式去噪作用，仅将概率质量集中于最相关词元。在开放词汇任务（密集语义分割：Pascal VOC、Pascal Context、ADE20K；细粒度检索：FG-OVD）上评估发现，注意力稀疏化带来的增益与基线注意力在目标类别外的分布范围成正比。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) relies on softmax-based self-attention, a strictly positive distribution that assigns probability mass to every pair of tokens-even semantically irrelevant ones. While these dense softmax weights are effective for gathering broad context during pre-training, they spread attention across many low-salience tokens, producing noise that obscures the fine-grained, spatially localized cues required for dense, open-vocabulary prediction. We study an inference-time substitution of the row-wise softmax in the final visual self-attention layers with the $α$-entmax transform, applied across both the standard query-key attention and self-correlation variants. Because entmax applies a data-dependent threshold that maps low scores exactly to zero, it acts as an implicit denoiser, zeroing contextually irrelevant dependencies while redistributing mass onto the most relevant tokens. We evaluate on open-vocabulary tasks-dense semantic segmentation (Pascal VOC, Pascal Context, ADE20K) and fine-grained retrieval (FG-OVD)-and find the gain from attention sparsification is proportional to how much the baseline attention spreads off the target class.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。