通过最小化注意力熵提升模型在分布偏移下的鲁棒性。
LookSharp: Attention Entropy Minimization for Test-Time Adaptation
- 聚焦最后一层的CLS到图像块注意力，最小化其熵值
- 在ImageNet-C上显著降低错误率，且不损害干净数据表现
- 可与输出熵最小化结合，适合部署时自适应场景
测试时自适应（TTA）在推理阶段更新模型以缓解分布偏移带来的误差。尽管对输出分布进行熵最小化已被证明有效，本文研究了利用Transformer中中间注意力分布作为优化目标。提出LookSharp方法，通过最小化最终层的CLS-to-patch注意力熵，引导模型在分布偏移数据上保持集中注意力。实验表明，该方法显著提升了ImageNet-C上的鲁棒性，且与输出熵最小化互补，同时在干净数据上保持性能。
原文摘要 · Abstract (English)
Test-time adaptation (TTA) updates models during inference to reduce error on distribution shifts. While entropy minimization over the output distribution has proven effective as a TTA loss, we study using the intermediate distributions computed by transformers in the attention mechanism. We propose LookSharp, which minimizes the entropy of CLS-to-patch attention in the final layer as a novel TTA objective, encouraging the model to maintain focused attention on shifted data. We demonstrate that attention entropy minimization improves robustness on ImageNet-C. We also show that it is complementary to output entropy minimization and maintains performance on clean data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。