用视觉噪声引导的上下文蒸馏,让多模态大模型安全删除敏感记忆。
Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning

- 通过视觉扰动+文本上下文干预,动态生成去记忆教师分布。
- 在不重训练前提下,遗忘集ROUGE-L降低0.371,保留集仅降0.055。
- 无需外部教师或标注,适合保护隐私的多模态应用。
多模态大语言模型(MLLM)在视觉-语言任务中取得显著进展,但可能记忆并暴露敏感或受限知识,引发隐私与安全风险。机器遗忘(MU)提供了一种无需从头训练即可移除特定不良知识的可行方案,同时保持模型通用能力。然而,有效实现MLLM的遗忘仍具挑战性。现有基于训练的方法常难以平衡遗忘效果与模型性能;而无训练方法如上下文遗忘虽能保留模型能力,却无法在参数层面清除记忆,仍易受逆向攻击。尤其在多模态场景中,视觉输入可提供强条件信号,诱发不良输出。为此,我们提出视觉噪声引导的上下文蒸馏(VGID),一种基于蒸馏的MLLM遗忘框架。VGID通过双模态干预——结合视觉扰动与文本上下文遗忘——从冻结的基础模型中动态构建去记忆导向的教师分布。该干预诱导的分布作为蒸馏教师信号,指导学生模型实现参数级遗忘,无需外部教师模型或显式不良响应标注。实验表明,VGID在保持良好模型性能的同时实现强遗忘效果:在代表性设置下,遗忘集ROUGE-L下降0.371,保留集仅下降0.055。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved remarkable progress on vision-language tasks, but they may also memorize and expose sensitive or restricted knowledge, raising concerns about privacy and broader safety risks. Machine Unlearning (MU) provides a promising way to remove targeted undesirable knowledge from trained models without retraining from scratch while preserving general model utility. Nevertheless, effective unlearning in MLLMs remains particularly challenging. Existing training-based methods often struggle to balance unlearning effectiveness and model utility. In contrast, training-free methods such as in-context unlearning preserve model utility by avoiding parameter updates, but they do not remove memorized knowledge at the parameter level and may remain vulnerable to reverse-engineering attacks. More importantly, in-context unlearning is insufficient in multimodal settings, where visual inputs can provide strong conditioning signals and induce undesirable outputs. To address these challenges, we propose Visual-Noise Guided In-Context Distillation (VGID), a distillation-based framework for MLLM unlearning. VGID dynamically constructs an unlearning-oriented teacher distribution from the frozen base model through dual-modal intervention that combines visual perturbation with textual in-context unlearning. The resulting intervention-induced distribution serves as a teacher signal for distillation, guiding the student model toward parameter-level unlearning without requiring external teacher models or explicit undesirable response annotations. Experimental results show that VGID achieves strong unlearning effectiveness while preserving competitive model utility, reducing forget set ROUGE-L by 0.371 with only a 0.055 drop in retain set ROUGE-L in a representative setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。