通过双重评估提升视觉语言模型的抗攻击能力，兼顾鲁棒性与推理速度。
SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
- 用稳定性与适用性双指标筛选增强视图，指导自适应和推理。
- 在多个数据集上实现更强鲁棒性，且吞吐量损失更小。
- 适合追求高效高鲁棒性的实际部署场景使用。
视觉语言模型（如CLIP）虽具备强大零样本识别能力，但在对抗扰动下仍十分脆弱。现有测试时自适应防御方法通过大量增强视图提升鲁棒性，但导致显著延迟，并存在鲁棒性与吞吐量的权衡。为此，本文提出稳定性与适用性引导的测试时提示调优（SS-TPT），通过两个互补评分评估每张增强视图的质量：(1) 稳定性，衡量对弱增强的预测不变性；(2) 适用性，衡量视图间特征空间密度。这两个评分（SS）通过SS引导的一致性损失和加权预测，强化可信视图，抑制受损视图。大量实验表明，SS-TPT显著优于先前最先进方法，在多种数据集及不同视图数量下均实现更优的鲁棒性-吞吐量平衡，展现出强实用性与通用性。代码已开源：https://github.com/sunoh-kim/SS-TPT。
原文摘要 · Abstract (English)
Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations. Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown and a clear robustness-throughput trade-off. To address this challenge, we present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), evaluating the quality of each augmented view via two complementary scores: (1) stability, measuring prediction invariance to weak augmentations, and (2) suitability, measuring feature-space density among views. These stability and suitability (SS) scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones. Extensive experiments demonstrate that SS-TPT significantly outperforms prior state-of-the-art methods, achieving superior robustness-throughput trade-offs across diverse datasets and varying numbers of views, thereby demonstrating both strong practicality and generality. Our code is available at https://github.com/sunoh-kim/SS-TPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。