T-VSS在测试时直接修正视觉特征，提升视觉语言模型抗干扰能力。
T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Models
- 在视觉特征空间构建低秩子空间，直接修正受攻击特征
- 通过加权熵最小化学习共享特征修正，提升预测稳定性
- 轻量高效，适合对鲁棒性要求高的实际应用
视觉语言模型（VLMs）具备强大的零样本识别能力，但对对抗扰动仍高度敏感。现有测试时自适应方法虽能提升鲁棒性，却未直接调整被破坏的视觉表征。提示法仅优化可学习文本提示，输入空间方法则在测试时优化像素或填充，路径间接且计算成本高。本文提出测试时视觉子空间引导（T-VSS），一种轻量级防御机制，直接在视觉特征空间进行测试时适应。T-VSS首先从多视角特征残差中构建以受攻击图像为锚点的样本专属低秩子空间，然后在该子空间内通过可靠性加权熵最小化学习共享特征修正。通过将适应限制在紧凑的视觉几何结构中，T-VSS将受攻击特征导向更稳定、更具判别性的预测，避免了全空间噪声更新。在细粒度分类、ImageNet及ImageNet-OOD基准上的实验表明，T-VSS在保持良好干净准确率的同时显著提升对抗鲁棒性，且效率优于先前测试时自适应方法。
原文摘要 · Abstract (English)
Vision-language models (VLMs) achieve strong zero-shot recognition, but they remain highly vulnerable to adversarial perturbations. Recent test-time adaptations improve robustness without retraining, but they do not directly adapt the corrupted visual representation itself. Prompt-based methods adapt the learnable text prompts, while input-space methods optimize pixels or padding at test time. These approaches can improve predictions, but they do so through an indirect and expensive optimization path. We propose Test-time Visual Subspace Steering (T-VSS), a lightweight defense that performs test-time adaptation directly in the visual feature space. T-VSS first builds a sample-specific low-rank subspace from multi-view feature residuals anchored at the attacked image. It then learns a shared feature correction within this subspace using reliability-weighted entropy minimization. By constraining adaptation to a compact visual geometry, T-VSS steers attacked features toward more stable and discriminative predictions while avoiding noisy full-space updates. Experiments on fine-grained, ImageNet, and ImageNet-OOD benchmarks show that T-VSS improves adversarial robustness while maintaining competitive clean accuracy and better efficiency than prior test-time adaptations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。