在CLIP嵌入空间中用增强漂移纠正对抗扰动，显著提升图像文本匹配鲁棒性。
ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
- 在CLIP嵌入空间通过增强扰动计算语义恢复方向
- 平均比现有最优方法提升近50%的抗攻击能力
- 计算开销极低，适用于极端和自适应攻击场景
尽管在零样本图像-文本匹配中表现卓越，CLIP仍对图像上的对抗扰动高度敏感。由于对抗微调成本过高，近期研究探索了多种测试时防御策略，但其鲁棒性仍有限。本文重新审视该问题，提出一种简单而有效的方法：基于增强的测试时对抗修正（ATAC）。该方法直接在CLIP的嵌入空间操作，通过计算增强引起的漂移向量，推断语义恢复方向，并根据这些潜在漂移的角一致性修正嵌入表示。在多个基准测试中，ATAC始终表现出极高的鲁棒性，平均性能比先前最先进方法高出近50%，且计算开销极小。此外，ATAC在非常规和极端设置下仍保持顶尖鲁棒性，甚至对自适应攻击也展现出非平凡的防御效果。结果表明，ATAC是一种高效的新范式，适用于CLIP嵌入空间的测试时对抗防御。代码已开源：https://github.com/kylin0421/ATAC。
原文摘要 · Abstract (English)
Despite its remarkable success in zero-shot image-text matching, CLIP remains highly vulnerable to adversarial perturbations on images. As adversarial fine-tuning is prohibitively costly, recent works explore various test-time defense strategies; however, these approaches still exhibit limited robustness. In this work, we revisit this problem and propose a simple yet effective strategy: Augmentation-based Test-time Adversarial Correction (ATAC). Our method operates directly in the embedding space of CLIP, calculating augmentation-induced drift vectors to infer a semantic recovery direction and correcting the embedding based on the angular consistency of these latent drifts. Across a wide range of benchmarks, ATAC consistently achieves remarkably high robustness, surpassing that of previous state-of-the-art methods by nearly 50\% on average, all while requiring minimal computational overhead. Furthermore, ATAC retains state-of-the-art robustness in unconventional and extreme settings and even achieves nontrivial robustness against adaptive attacks. Our results demonstrate that ATAC is an efficient method in a novel paradigm for test-time adversarial defenses in the embedding space of CLIP. Code is available at: https://github.com/kylin0421/ATAC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。