小模型注意力更准，大模型表征更强，A²巧妙结合两者提升定位能力。
$A^2$: Smaller Self-Supervised ViTs Localize Better than Larger Ones
- 用小模型找关注点，大模型提取特征，分离定位与表征任务。
- 在5个基准上性能媲美端到端训练方法，分布外场景下优势明显。
- 无需标注、无需训练，全靠预训练模型直接组合，部署极简。
鲁棒的视觉分类常依赖于定位图像中的主要前景物体并忽略上下文干扰。令人惊讶的是，我们发现较小的自监督ViT生成的注意力图比大型ViT更能准确地定位前景物体。然而，仍需使用大ViT，因为其能从每个图像块中提取更丰富的表示。为兼顾两者优势，我们提出A²：一种简单方法，将‘看哪里’（小注意力模型）与‘提取什么’（大嵌入模型）解耦——先用小模型定位注意力峰值区域并裁剪，再用大模型对裁剪区域进行嵌入。A²完全使用预训练特征，无需类别标签，也不需要针对数据集的注意力或主干网络训练。在5个基准测试中，A²表现与匹配骨干网络的损失级方法（如DFR）相当，并在更强的分布外转移场景下优于端到端注意力训练方法。
原文摘要 · Abstract (English)
Robust visual classification often depends on localizing the main foreground objects in an image while ignoring contextual distractors. Surprisingly, we find that the attention maps of smaller self-supervised ViTs localize foreground objects better than those of larger ViTs. However, we still need large ViTs, because they extract richer representations from each patch. To get the best of both worlds, good localization and rich representations, we propose $A^2$, a simple method that leverages this inverse scaling finding by decoupling where to look (a small attention model) from what to extract (a large embedding model): we crop around the attention peaks of a small model and embed the crops with a larger model. $A^2$ uses entirely pretrained features, requires no group labels, and does not require per-dataset attention or backbone training. Across 5 benchmarks, $A^2$ is competitive with backbone-matched loss-level methods like DFR, and outperforms end-to-end attention training under stronger distribution shifts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。