让视觉语言模型在推理时自适应调整防御强度,提升抗攻击能力。
ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Models

- 根据输入特征漂移程度动态决定防御强度
- 在12个数据集上对多种攻击均有效,且保持原始准确率
- 无需训练,适用于不同攻击强度和数据分布
无训练的测试时防御为CLIP类视觉-语言模型提供了一种实用的对抗鲁棒性增强方式,而无需修改预训练模型。然而,现有方法的修正强度通常固定在一个狭窄的攻击预算范围内,尽管推理时攻击预算未知,且不同样本所需修正程度各异。我们发现这种不匹配会导致防御效果随攻击增强而急剧下降。为此提出ReACT-CLIP,一种响应感知的测试时防御机制,可分别判断每个输入应进行多强的修正以及是否需要干预。关键观察是:低噪声与高噪声探测下CLIP视觉特征漂移的相对变化,可作为样本级修正需求的连续代理。ReACT-CLIP将该相对跨噪声漂移映射到用于构建稳定噪声平均特征锚点的高斯噪声尺度,使修正范围能自适应输入。为进一步判断是否需干预,我们观察到干净输入在弱空间增强下类别概率分布稳定,而对抗输入则表现出更大波动。ReACT-CLIP利用预测不稳定性分数(通过Jensen-Shannon散度计算)结合相对跨噪声漂移,形成防御干预评分。ReACT-CLIP无需模型或提示训练,其修正强度映射仅需一次校准并在不同数据集和攻击预算间通用。在12个下游数据集及ImageNet与其分布偏移变体上,该方法对多种攻击类型和强度均带来显著鲁棒性提升,同时基本保持原始准确率。
原文摘要 · Abstract (English)
Training-free test-time defenses offer a practical way to improve the adversarial robustness of CLIP-style vision--language models without modifying the pretrained model. However, their correction strength is typically fixed for a narrow range of attack budgets, even though the attack budget is unknown at inference and the required correction varies across samples. We show that this mismatch causes existing defenses to degrade sharply as attacks strengthen. We introduce ReACT-CLIP, a response-conditioned test-time defense that separately determines how strongly each input should be corrected and whether defensive intervention is necessary. Our key observation is that the relative increase in CLIP visual-feature drift between low- and high-noise probes provides a graded, sample-specific proxy for correction demand. ReACT-CLIP maps this relative cross-noise drift to the Gaussian noise scale used to construct a stable, noise-averaged feature anchor, enabling the corrective reach to adapt to each input. To determine whether intervention is necessary, we further observe that clean inputs retain stable class-probability distributions under weak spatial augmentations, whereas adversarial inputs exhibit greater variation. ReACT-CLIP quantifies this variation using a prediction-instability score computed by Jensen--Shannon divergence and combines it with relative cross-noise drift to form the defensive intervention score. ReACT-CLIP requires no model or prompt training, and its correction-strength mapping is calibrated once and fixed across datasets and attack budgets. Across 12 downstream datasets, as well as ImageNet and its distribution-shifted variants, ReACT-CLIP delivers substantial robustness gains across diverse attack types and strengths while largely preserving clean accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。