提出CAGE攻击，精准打击压缩版视觉语言模型的脆弱性
On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
- 设计与压缩推理对齐的攻击，无需知晓压缩机制
- 在多种压缩方法下均使模型鲁棒性显著下降
- 适合研究高效模型安全或对抗攻防的读者
视觉标记压缩广泛用于加速大型视觉-语言模型（LVLMs），通过剪枝或合并视觉标记实现，但其对抗鲁棒性尚未被深入研究。现有基于编码器的攻击无法充分暴露压缩后LVLM的脆弱性，原因在于优化与推理之间的不匹配：扰动在完整标记表示上优化，而推理却经过标记压缩瓶颈。为解决这一问题，我们提出压缩对齐攻击（CAGE），该方法在不假设已知部署压缩机制或其标记预算的情况下，将扰动优化与压缩推理对齐。CAGE结合两项机制：(i) 期望特征破坏，将失真集中在可能在不同预算下存活的标记上；(ii) 排名失真对齐，主动将标记失真与排名得分对齐，以促进高度失真的证据保留。在多种代表性即插即用压缩机制和数据集上的实验表明，CAGE始终比基线获得更低的鲁棒准确率。本工作表明，忽略压缩的鲁棒性评估可能过于乐观，呼吁对高效LVLM进行压缩感知的安全评估与防御。
原文摘要 · Abstract (English)
Visual token compression is widely used to accelerate large vision-language models (LVLMs) by pruning or merging visual tokens, yet its adversarial robustness remains unexplored. We show that existing encoder-based attacks cannot fully disclose the robustness vulnerabilities of compressed LVLMs, due to an optimization-inference mismatch: perturbations are optimized on the full-token representation, while inference is performed through a token-compression bottleneck. To address this gap, we propose the Compression-AliGnEd attack (CAGE), which aligns perturbation optimization with compression inference without assuming access to the deployed compression mechanism or its token budget. CAGE combines (i) expected feature disruption, which concentrates distortion on tokens likely to survive across plausible budgets, and (ii) rank distortion alignment, which actively aligns token distortions with rank scores to promote the retention of highly distorted evidence. Across diverse representative plug-and-play compression mechanisms and datasets, our results show that CAGE consistently achieves lower robust accuracy than the baseline. This work highlights that robustness assessments ignoring compression can be overly optimistic, calling for compression-aware security evaluation and defenses for efficient LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。