研究视觉语言模型中跨模态遗忘的不对称性,提出新方法提升遗忘鲁棒性。
One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models

- 通过影响引导策略聚焦关键模块,实现更有效的跨模态遗忘
- 在强融合架构中使遗忘传递差距减少超50%,攻击成功率接近零
- 适用于需要安全删除敏感信息的视觉语言模型部署场景
机器遗忘广泛用于从大语言模型中移除有害知识。然而,现代视觉语言模型(VLMs)同时处理文本和视觉输入,引发一个根本性安全问题:在一个模态中进行遗忘,是否能传递到另一个模态?我们首次系统性地研究了三种VLM架构(LLaVA-1.5、InstructBLIP、IDEFICS)中的双向跨模态遗忘转移。结果发现,遗忘确实会跨模态传递,但具有不对称性和不完整性。在某些情况下,文本遗忘能有效传递至视觉模态;但在文字排版攻击下,先前已遗忘的知识可被轻易恢复,表明遗忘深度不足。为此,我们提出 extsc{CrossInf},一种基于影响引导的缓解策略。该方法聚焦于对跨模态泛化贡献最大的变压器块,显著缩小遗忘传递差距,在强融合架构中降低超过一半,并保持模型性能。同时,在排版攻击下将攻击成功率降至近零。我们还通过三位标注员的人类评估验证结果(κ=0.77)。最后,利用中心核对齐(CKA)分析浅层遗忘现象,揭示了转移行为与鲁棒性局限的内在机制。
原文摘要 · Abstract (English)
Machine unlearning is widely used to remove hazardous knowledge from large language models. Modern Vision-Language Models (VLMs), however, process both text and visual inputs, raising a fundamental security question: does unlearning in one modality transfer to the other? We present the first systematic, bidirectional study of cross-modal unlearning transfer across three VLM architectures: LLaVA-1.5 (MLP projection), InstructBLIP (Q-Former), and IDEFICS (gated cross-attention). We find that unlearning transfers across modalities, but the transfer is asymmetric and incomplete. In some cases, text unlearning strongly transfers to vision. However, this robustness is not preserved under typographic attacks that manipulate the visual presentation of text. Under such attacks, previously unlearned knowledge can be readily recovered, indicating shallow unlearning. To address the transfer gap and shallow robustness, we propose \textsc{CrossInf}, an influence-guided mitigation strategy. Motivated by the observation that different model components contribute unequally to cross-modal transfer, \textsc{CrossInf} focuses unlearning on transformer blocks that most influence cross-modal generalization. It reduces the transfer gap by more than half in architectures with strong fusion, while preserving model utility. It also improves robustness under typographic attacks, reducing the attack success rate to near zero. We further conduct human evaluation with three annotators ($κ{=}0.77$) to validate our findings. Finally, we analyze shallow unlearning using Centered Kernel Alignment (CKA), providing insights into the observed transfer behavior and robustness limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。