通过能量引导的测试时变换，提升大模型对抗攻击下的可靠性。
A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models
- 基于输入样本能量最小化，设计轻量无训练防御方法
- 在图像描述与视觉问答任务中显著增强大模型鲁棒性
- 理论可证明其有效性，适合部署于真实场景
尽管多模态模型和大视觉语言模型（LVLM）发展迅速，但它们仍极易受到对抗扰动影响，严重威胁实际应用中的可靠性。虽然对抗训练是主流的鲁棒建模范式，但测试时变换（TTT）已成为一种有前景的推理阶段增强鲁棒性的策略。为此，我们提出能量引导的测试时变换（ET3），一种轻量、无需训练的防御方法，通过最小化输入样本的能量来增强分类性能。该方法基于理论证明，在合理假设下其变换能成功实现分类。大量实验表明，ET3在分类器、CLIP零样本分类，以及图像描述和视觉问答等任务中，显著提升了LVLM的对抗鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Despite the rapid progress in multimodal models and Large Visual-Language Models (LVLM), they remain highly susceptible to adversarial perturbations, raising serious concerns about their reliability in real-world use. While adversarial training has become the leading paradigm for building models that are robust to adversarial attacks, Test-Time Transformations (TTT) have emerged as a promising strategy to boost robustness at inference. In light of this, we propose Energy-Guided Test-Time Transformation (ET3), a lightweight, training-free defense that enhances the robustness by minimizing the energy of the input samples. Our method is grounded in a theory that proves our transformation succeeds in classification under reasonable assumptions. We present extensive experiments demonstrating that ET3 provides a strong defense for classifiers, zero-shot classification with CLIP, and also for boosting the robustness of LVLMs in tasks such as Image Captioning and Visual Question Answering. Code is available at github.com/OmnAI-Lab/Energy-Guided-Test-Time-Defense .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。