arXiv:2603.12989cs.CVcs.CR2026-03被引 5

提出测试时净化注意力的防御方法,有效抵御视觉语言模型后门攻击。

Test-Time Attention Purification for Backdoored Large Vision Language Models

  • 发现后门通过异常跨模态注意力劫持触发,而非低层视觉特征。
  • 在多个数据集上对齐攻击类型,净化后模型准确率提升15%以上。
  • 无需训练、可即插即用,适合部署阶段快速防御。

尽管大型视觉语言模型(LVLM)具备强大多模态性能,但在微调过程中仍易受后门攻击影响,攻击者通过在训练数据中插入带触发器的样本,植入可在测试时恶意激活的行为。现有防御方法通常依赖于使用干净数据重新训练被污染的参数(如适配器或LoRA模块),计算成本高且常导致模型性能下降。本文提供了对LVLM后门行为的新机制理解:触发器并非通过低层视觉模式影响预测,而是通过异常的跨模态注意力重分配——携带触发器的视觉标记会劫持本应属于文本上下文的注意力,我们称之为‘注意力劫持’。受此启发,我们提出CleanSight,一种无需训练、纯测试时运行的即插即用式防御。CleanSight (i) 基于选定跨模态融合层中的视觉-文本注意力相对比例检测中毒输入,(ii) 通过选择性剪枝高注意力可疑视觉标记来净化输入,从而消除后门激活。大量实验表明,CleanSight在多种数据集和攻击类型下显著优于现有基于像素的净化防御方法,同时在干净与中毒样本上均保持模型实用性。

原文摘要 · Abstract (English)

Despite the strong multimodal performance, large vision-language models (LVLMs) are vulnerable during fine-tuning to backdoor attacks, where adversaries insert trigger-embedded samples into the training data to implant behaviors that can be maliciously activated at test time. Existing defenses typically rely on retraining backdoored parameters (e.g., adapters or LoRA modules) with clean data, which is computationally expensive and often degrades model performance. In this work, we provide a new mechanistic understanding of backdoor behaviors in LVLMs: the trigger does not influence prediction through low-level visual patterns, but through abnormal cross-modal attention redistribution, where trigger-bearing visual tokens steal attention away from the textual context - a phenomenon we term attention stealing. Motivated by this, we propose CleanSight, a training-free, plug-and-play defense that operates purely at test time. CleanSight (i) detects poisoned inputs based on the relative visual-text attention ratio in selected cross-modal fusion layers, and (ii) purifies the input by selectively pruning the suspicious high-attention visual tokens to neutralize the backdoor activation. Extensive experiments show that CleanSight significantly outperforms existing pixel-based purification defenses across diverse datasets and backdoor attack types, while preserving the model's utility on both clean and poisoned samples.

后门防御视觉语言模型注意力净化测试时防御

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。