提出选择性适配，只对必要样本进行测试时调整，提升视觉语言模型鲁棒性。
To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models
- 根据多视角预测相似度判断是否需要适配，避免无效或有害调整
- 跳过近85%的适配操作，整体准确率仍保持甚至提升
- 适合关注模型推理稳定性与效率的研究者
测试时适配（TTA）已成为视觉语言模型应对推理阶段分布偏移的重要策略。我们对适配前后每一样本的预测进行分析，发现现有方法存在两种失效模式：适配效果微乎其微，预测结果无变化；更严重的是，适配可能使原本正确的预测变为错误。这引发一个关键问题：能否识别并跳过这些无效或有害的适配？本文提出选择性适配新任务，旨在判断测试样本是否应进行适配。为此，我们设计跨增强相似度（CAS），仅当不同增强视图下的预测相似度较低时才执行适配。值得注意的是，CAS不仅保持了性能，甚至在某些情况下提升了准确率，同时跳过了近85%的适配过程。我们希望激励更多研究者探索这一方向，超越当前基线。代码已开源：https://github.com/sirujiang/selective-adaptation。
原文摘要 · Abstract (English)
Test-time adaptation (TTA) has emerged as a prominent strategy for adapting vision-language models to distribution shifts during inference. We conduct a per-sample analysis of model predictions before and after adaptation, and observe two failure modes in existing TTA methods that echo previous work. Adaptations are frequently negligible, yielding no change in the model's predictions, and more severely, they can be detrimental by flipping previously correct predictions to incorrect ones. This naturally raises a question: Can we identify and skip such negligible or harmful adaptations? In this work, we introduce a new problem of selective adaptation, which aims to determine whether a given test sample should undergo adaptation or be skipped. To this end, we propose Cross-Augmentation Similarity (CAS), a simple baseline that performs adaptation only when predictions across augmented views exhibit low similarity. Notably, CAS not only preserves but in some cases improves overall accuracy, even when skipping nearly 85% of the adaptation process. We hope other researchers will explore this new direction and surpass the performance of our baseline. Our code is available at https://github.com/sirujiang/selective-adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。