通过跨模态协同优化,提升多模态大模型攻击的泛化能力。
Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models
- 融合视觉与文本语义表示,实现跨范式联合扰动优化。
- 在多个基准上超越现有方法,对开源与闭源模型均有效。
- 适合研究模型鲁棒性或对抗攻击的学者参考。
多模态大语言模型(MLLMs)的快速发展推动了下游应用的进步,但也暴露了严重的可迁移对抗漏洞。现有攻击方法通常依赖单一学习范式下的代理模型,并在各自特征空间中独立优化,限制了特征表示的丰富性,压缩了搜索空间,阻碍了对抗扰动的多样性。为此,我们提出一种新型多范式协同攻击框架(MPCAttack),以增强对抗样本对MLLMs的可迁移性。该框架通过多范式协同优化(MPCO)策略,聚合图像与文本的语义表示,在融合特征上进行联合对抗优化。通过在多范式特征间执行对比匹配,MPCO自适应平衡不同范式表示的重要性,引导全局扰动优化,有效缓解表示偏差。大量实验表明,MPCAttack在多个基准上均优于现有最先进方法,对开源与闭源MLLMs在目标攻击和非目标攻击中均表现出持续优势。代码已开源:https://github.com/LiYuanBoJNU/MPCAttack。
原文摘要 · Abstract (English)
The rapid progress of Multi-Modal Large Language Models (MLLMs) has significantly advanced downstream applications. However, this progress also exposes serious transferable adversarial vulnerabilities. In general, existing adversarial attacks against MLLMs typically rely on surrogate models trained within a single learning paradigm and perform independent optimisation in their respective feature spaces. This straightforward setting naturally restricts the richness of feature representations, delivering limits on the search space and thus impeding the diversity of adversarial perturbations. To address this, we propose a novel Multi-Paradigm Collaborative Attack (MPCAttack) framework to boost the transferability of adversarial examples against MLLMs. In principle, MPCAttack aggregates semantic representations, from both visual images and language texts, to facilitate joint adversarial optimisation on the aggregated features through a Multi-Paradigm Collaborative Optimisation (MPCO) strategy. By performing contrastive matching on multi-paradigm features, MPCO adaptively balances the importance of different paradigm representations and guides the global perturbation optimisation, effectively alleviating the representation bias. Extensive experimental results on multiple benchmarks demonstrate the superiority of MPCAttack, indicating that our solution consistently outperforms state-of-the-art methods in both targeted and untargeted attacks on open-source and closed-source MLLMs. The code is released at https://github.com/LiYuanBoJNU/MPCAttack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。