arXiv:2605.15584cs.CV2026-05

提出无需训练的对抗鲁棒性方法,显著提升视觉语言模型安全性

AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models

论文配图:AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models
图 1 · 摘自论文原文
  • 通过选择可靠数据增强作为几何锚点,动态修正特征方向
  • 在8个细粒度数据集上平均鲁棒准确率提升44.4%,推理延迟降低10倍
  • 适合部署场景下对效率与安全有要求的视觉语言模型应用

如CLIP的视觉语言模型展现出强大的零样本迁移能力,但其对难以察觉的对抗扰动仍极为敏感。尽管测试时防御为已部署模型提供了实用方案,现有方法通常依赖推理时的梯度优化,带来显著计算开销。本文重新审视数据增强在CLIP鲁棒性中的作用,发现不同增强效果不均:特定增强能持续提供与正确类别语义对齐的鲁棒几何线索。基于此,提出训练无关的自适应测地线校正(AGC)机制,无需参数更新。AGC识别一个可靠增强作为几何锚点,将输入特征向其校正,采用自适应步长平衡鲁棒性与原始准确率。AGC在8个细粒度数据集和3种CLIP骨干网络上表现优异,相比最先进基线平均鲁棒准确率提升44.4%,推理延迟降低10倍。研究揭示了CLIP特征的基本几何特性,为多模态部署提供高效可靠的鲁棒范式。

原文摘要 · Abstract (English)

Vision-language models like CLIP have demonstrated remarkable zero-shot transfer capabilities. However, their susceptibility to imperceptible adversarial perturbations remains a critical security concern. While test-time defenses offer a pragmatic solution for deployed models, existing approaches typically rely on gradient-based optimization during inference, incurring significant computational overhead. In this paper, we revisit the role of data augmentation in CLIP robustness and observe that augmentations are not equally effective: specific augmentations consistently provide robust geometric cues that align with correct class semantics in the hyperspherical feature space. Based on this, we propose Adaptive Geodesic Correction (AGC), a training-free defense mechanism that requires no parameter updates. AGC identifies a reliable augmentation as a geometric anchor and corrects the input feature towards it, utilizing an adaptive step size to balance robustness against clean accuracy preservation. AGC achieves superior performance across eight fine-grained datasets and three CLIP backbones, improving average robust accuracy by 44.4\% over state-of-the-art baseline while delivering a 10$\times$ reduction in inference latency. Our findings reveal a fundamental geometric property of CLIP features, offering a highly efficient and effective paradigm for robust multimodal deployment.

对抗鲁棒性视觉语言模型测试时防御几何校正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。