提出一种高效测试时防御方法，提升CLIP对未知攻击的鲁棒性。
Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP
- 基于频谱引导对比学习，自适应修复输入特征分布
- 在16个基准上平均提升18.1%对抗鲁棒性，延迟低
- 适用于多种视觉任务，无需重新训练
视觉-语言模型（如CLIP）展现出强大的零样本泛化能力，但仍极易受到对抗样本（AEs）攻击。尽管测试时防御方法前景广阔，但现有方法难以应对强攻击，且常伴随高推理延迟和任务特异性。我们首先分析对抗样本的内在特性，发现其在逐步频率衰减下表现出严重特征不一致，根源在于模型固有的频谱偏差。基于此，我们提出一种高效的测试时防御方法——对比频谱修正（CSR）。CSR通过优化一个自适应的修正扰动，在频谱引导的对比目标下将输入重对齐至自然流形。大量实验表明，该方法在16个分类基准上平均比当前最优方法提升18.1%，且推理开销小。此外，CSR在多种视觉任务中均具广泛适用性。代码已公开于https://github.com/Summu77/CSR。
原文摘要 · Abstract (English)
Vision-language models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, yet remain highly vulnerable to adversarial examples (AEs). While test-time defenses are promising, existing methods fail to provide sufficient robustness against strong attacks and are often hampered by high inference latency and task-specific applicability. To address these limitations, we start by investigating the intrinsic properties of AEs, which reveals that AEs exhibit severe feature inconsistency under progressive frequency attenuation. We further attribute this to the model's inherent spectral bias. Leveraging this insight, we propose an efficient test-time defense named Contrastive Spectral Rectification (CSR). CSR optimizes a rectification perturbation to realign the input with the natural manifold under a spectral-guided contrastive objective, which is applied input-adaptively. Extensive experiments across 16 classification benchmarks demonstrate that CSR outperforms the SOTA by an average of 18.1% against strong APGD with modest inference overhead. Furthermore, CSR exhibits broad applicability across diverse visual tasks. Code is available at https://github.com/Summu77/CSR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。