arXiv:2505.11895cs.CV2025-05

提出高效校准框架,提升多模态模型对抗攻击下的鲁棒性。

Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration

  • 用对抗样本训练特定模态的投影头,不改动主干网络
  • 在六种模态上实现最高47.3%的鲁棒性提升
  • 仅需不到1%可训练参数,兼容现有大模型

统一多模态编码器将多种模态对齐至共享表示空间,支持跨模态任务。尽管能力突出,其在对抗扰动下的鲁棒性仍缺乏研究,这对安全敏感应用至关重要。本文首次系统研究统一多模态编码器的对抗脆弱性,发现轻微对抗扰动即导致所有模态性能显著下降,非视觉输入(如音频、点云)尤为脆弱,视觉输入(图像、视频)也明显退化。为此,我们提出一种高效对抗校准框架,在不修改预训练编码器或语义中心的前提下提升跨模态鲁棒性。方法通过仅在对抗样本上训练模态特定的投影头,保持主干与嵌入冻结。探索三种训练目标:固定中心交叉熵、干净-对抗样本L2对齐、干净-对抗信息瓶颈(InfoNCE),并引入正则化策略确保攻击下模态一致性对齐。在六种模态和三个Bind-style模型上的实验表明,该方法在ε=4/255时对抗鲁棒性提升高达47.3%,同时保持甚至提升原始零样本与检索性能,可训练参数少于1%。

原文摘要 · Abstract (English)

Recent unified multi-modal encoders align a wide range of modalities into a shared representation space, enabling diverse cross-modal tasks. Despite their impressive capabilities, the robustness of these models under adversarial perturbations remains underexplored, which is a critical concern for safety-sensitive applications. In this work, we present the first comprehensive study of adversarial vulnerability in unified multi-modal encoders. We find that even mild adversarial perturbations lead to substantial performance drops across all modalities. Non-visual inputs, such as audio and point clouds, are especially fragile, while visual inputs like images and videos also degrade significantly. To address this, we propose an efficient adversarial calibration framework that improves robustness across modalities without modifying pretrained encoders or semantic centers, ensuring compatibility with existing foundation models. Our method introduces modality-specific projection heads trained solely on adversarial examples, while keeping the backbone and embeddings frozen. We explore three training objectives: fixed-center cross-entropy, clean-to-adversarial L2 alignment, and clean-adversarial InfoNCE, and we introduce a regularization strategy to ensure modality-consistent alignment under attack. Experiments on six modalities and three Bind-style models show that our method improves adversarial robustness by up to 47.3 percent at epsilon = 4/255, while preserving or even improving clean zero-shot and retrieval performance with less than 1 percent trainable parameters.

多模态对抗鲁棒性高效校准大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。