用张量分解提升视觉语言模型抗攻击能力，无需重训练
Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks
- 通过张量分解重构视觉编码特征，滤除对抗噪声
- 在Flickr30K上恢复12.3%性能损失，召回率从7.5%提至19.8%
- 低秩张量列车分解+小残差强度，适合即插即用
视觉语言模型（VLMs）在多模态理解中表现优异，但易受对抗攻击影响。现有防御方法通常需要昂贵的重新训练或显著的架构修改。本文提出一种轻量级防御机制，适用于任何预训练的VLM，无需重新训练。通过分解并重构视觉编码器表示，该方法能有效过滤对抗噪声，同时保留语义信息。在CLIP模型上对COCO和Flickr30K数据集的实验表明，该方法显著提升了鲁棒性：在Flickr30K上恢复了12.3%的性能损失，使Recall@1准确率从7.5%提升至19.8%；在COCO上恢复8.1%性能，准确率从3.8%提高到11.9%。分析显示，使用低秩（8-32）和低残差强度（α=0.1-0.2）的张量列车分解效果最佳。该方法为现有VLM提供了一种高效、即插即用的防御方案。
原文摘要 · Abstract (English)
Vision language models (VLMs) excel in multimodal understanding but are prone to adversarial attacks. Existing defenses often demand costly retraining or significant architecture changes. We introduce a lightweight defense using tensor decomposition suitable for any pre-trained VLM, requiring no retraining. By decomposing and reconstructing vision encoder representations, it filters adversarial noise while preserving meaning. Experiments with CLIP on COCO and Flickr30K show improved robustness. On Flickr30K, it restores 12.3\% performance lost to attacks, raising Recall@1 accuracy from 7.5\% to 19.8\%. On COCO, it recovers 8.1\% performance, improving accuracy from 3.8\% to 11.9\%. Analysis shows Tensor Train decomposition with low rank (8-32) and low residual strength ($α=0.1-0.2$) is optimal. This method is a practical, plug-and-play solution with minimal overhead for existing VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。