不重训模型，用推理时注意力工程提升视觉变换器聚类性能
Improving Image Clustering with Artifacts Attenuation via Inference-Time Attention Engineering
- 推理时通过分析QKV特征，识别并抑制多头注意力中的异常高值块
- 在多个数据集上显著提升聚类准确率，特征表达更丰富
- 适合希望零成本优化预训练视觉模型聚类能力的研究者
本文旨在提升预训练视觉变换器（ViT）模型，尤其是DINOv2，在图像聚类任务中的表现，且无需重新训练或微调。随着模型规模增大，多头注意力中出现高范数异常块，导致零样本聚类准确率下降。这些异常表现为注意力图中某些块的值远高于其他块。为此，我们提出推理时注意力工程（ITAE），在推理阶段操纵注意力函数：通过分析多头注意力中的一个查询-键-值（QKV）块识别异常，并衰减其对应注意力值。ITAE在多个数据集上均实现聚类准确率提升，使潜在空间特征更具表达性。研究结果表明，ITAE是一种有效降低预训练ViT模型中异常现象、提升聚类性能的实用方案，且无需重训练。
原文摘要 · Abstract (English)
The goal of this paper is to improve the performance of pretrained Vision Transformer (ViT) models, particularly DINOv2, in image clustering task without requiring re-training or fine-tuning. As model size increases, high-norm artifacts anomaly appears in the patches of multi-head attention. We observe that this anomaly leads to reduced accuracy in zero-shot image clustering. These artifacts are characterized by disproportionately large values in the attention map compared to other patch tokens. To address these artifacts, we propose an approach called Inference-Time Attention Engineering (ITAE), which manipulates attention function during inference. Specifically, we identify the artifacts by investigating one of the Query-Key-Value (QKV) patches in the multi-head attention and attenuate their corresponding attention values inside the pretrained models. ITAE shows improved clustering accuracy on multiple datasets by exhibiting more expressive features in latent space. Our findings highlight the potential of ITAE as a practical solution for reducing artifacts in pretrained ViT models and improving model performance in clustering tasks without the need for re-training or fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。