通过移除有害注意力头，提升CLIP图像表征质量
Not All Attention Heads Are What You Need: Refining CLIP's Image Representation with Attention Ablation
- 识别并抑制对表征有害的注意力头
- 跨模态检索召回率最高提升11.1%
- 无需额外计算开销，适合部署优化
本文研究CLIP图像编码器中注意力头的作用。基于可解释性分析，我们系统性发现部分分布于不同层的注意力头会损害最终表征质量。为此提出一种简单有效的注意力消融技术（AAT），通过直接修改注意力权重来抑制选定头。结合两种互补策略，AAT可在不同应用场景下高效定位并消除有害头，开销极低。实验表明，AAT在多个下游任务中持续提升性能，跨模态检索最高提升11.1%的召回率。结果表明，AAT能以几乎无额外推理成本有效优化大规模视觉语言模型，并产生与现有可解释性研究一致的语义模式。
原文摘要 · Abstract (English)
This paper investigates the role of attention heads in CLIP's image encoder. Building on interpretability studies, we conduct an exhaustive analysis and find that certain heads, distributed across layers, are detrimental to the resulting representations. To mitigate their impact, we propose a simple yet effective Attention Ablation Technique (AAT) that suppresses selected heads by directly manipulating their attention weights. By incorporating two complementary strategies tailored to different application scenarios, AAT enables the systematic identification and ablation of harmful heads with minimal overhead. Experiments show that AAT consistently improves downstream performance across diverse domains, boosting recall by up to 11.1% on cross-modal retrieval benchmarks. These results highlight that AAT can effectively refine large-scale VLMs with virtually no extra inference cost, while yielding semantically meaningful patterns that align with existing interpretability findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。