通过优化全局与局部特征对齐,提升对闭源多模态大模型的攻击转移性。
Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment
- 融合全局与局部特征对齐,利用最优传输优化细粒度匹配。
- 在多个模型上实现更强攻击转移性,尤其对闭源模型效果显著。
- 适合研究模型安全与对抗样本攻防的从业者使用。
多模态大语言模型(MLLM)仍易受可迁移对抗样本攻击。现有方法通常通过对齐全局特征(如CLIP的[CLS]标记)实现目标攻击,但忽视了嵌入在补丁标记中的丰富局部信息,导致对齐不充分、迁移能力有限,尤其在闭源模型上表现不佳。为此,我们提出基于特征最优对齐的目标可迁移对抗攻击方法FOA-Attack。在全局层面,引入基于余弦相似度的全局特征损失,对齐对抗样本与目标样本的粗粒度特征;在局部层面,利用聚类技术提取紧凑的局部模式,缓解冗余特征,并将局部特征对齐建模为最优传输(OT)问题,提出局部聚类最优传输损失以精炼细粒度对齐。此外,设计动态集成模型加权策略,自适应平衡多模型影响,进一步提升迁移性。大量实验表明,该方法在多种模型上均优于当前最先进方法,尤其在闭源MLLM上的迁移性能突出。代码已开源:https://github.com/jiaxiaojunQAQ/FOA-Attack。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples. While existing methods typically achieve targeted attacks by aligning global features-such as CLIP's [CLS] token-between adversarial and target samples, they often overlook the rich local information encoded in patch tokens. This leads to suboptimal alignment and limited transferability, particularly for closed-source models. To address this limitation, we propose a targeted transferable adversarial attack method based on feature optimal alignment, called FOA-Attack, to improve adversarial transfer capability. Specifically, at the global level, we introduce a global feature loss based on cosine similarity to align the coarse-grained features of adversarial samples with those of target samples. At the local level, given the rich local representations within Transformers, we leverage clustering techniques to extract compact local patterns to alleviate redundant local features. We then formulate local feature alignment between adversarial and target samples as an optimal transport (OT) problem and propose a local clustering optimal transport loss to refine fine-grained feature alignment. Additionally, we propose a dynamic ensemble model weighting strategy to adaptively balance the influence of multiple models during adversarial example generation, thereby further improving transferability. Extensive experiments across various models demonstrate the superiority of the proposed method, outperforming state-of-the-art methods, especially in transferring to closed-source MLLMs. The code is released at https://github.com/jiaxiaojunQAQ/FOA-Attack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。