让多模态大模型删掉特定视觉知识,不伤其他内容。
Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearning

- 用对比方法分离要删的视觉信息,引导特征到指定区域。
- 在保留知识的零空间内操作,减少对其他知识的干扰。
- 支持持续删除请求,适合真实场景长期使用。
机器遗忘的核心挑战在于平衡目标知识的消除与非目标知识的保留。在多模态大语言模型(MLLMs)中,这一挑战更为突出,因为知识被分为紧密交织的视觉和文本模态。本文提出一种MLLM遗忘方法,旨在删除目标视觉知识的同时,保留非目标视觉知识和全部文本知识。具体地,冻结LLM主干,仅微调视觉模块。首先,提出对比视觉遗忘(CVF)机制,将目标视觉知识与保留知识分离,引导目标视觉概念特征向特征空间中的特定区域移动。其次,识别与保留知识相关的零空间,并将遗忘过程约束于该空间内,显著缓解知识保留的退化。第三,将方法扩展至持续遗忘场景,支持顺序到达的遗忘请求。在多个基准上的大量实验表明,该方法在有效遗忘与稳健知识保留之间取得了良好平衡。
原文摘要 · Abstract (English)
The core challenge of machine unlearning is to strike a balance between target knowledge removal and non-target knowledge retention. In the context of Multimodal Large Language Models (MLLMs), this challenge becomes even more pronounced, as knowledge is further divided into visual and textual modalities that are tightly intertwined. In this paper, we introduce an MLLM unlearning approach that aims to forget target visual knowledge while preserving non-target visual knowledge and all textual knowledge. Specifically, we freeze the LLM backbone and achieve unlearning by fine-tuning the visual module. First, we propose a Contrastive Visual Forgetting (CVF) mechanism to separate target visual knowledge from retained visual knowledge, guiding the representations of target visual concepts toward appropriate regions in the feature space. Second, we identify the null space associated with retained knowledge and constrain the unlearning process within this space, thereby significantly mitigating degradation in knowledge retention. Third, beyond static unlearning scenarios, we extend our approach to continual unlearning, where forgetting requests arrive sequentially. Extensive experiments across diverse benchmarks demonstrate that our approach achieves a strong balance between effective forgetting and robust knowledge retention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。