通过路径编辑实现多模态模型精准删忆,兼顾遗忘效果与通用能力保留。
Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models
- 基于模态专属重要性评分定位跨模态关键神经路径。
- 在多模态任务中最高实现87.75%遗忘率,通用性能保留率达77.9%。
- 适合需隐私保护的多模态大模型场景,如医疗影像与文本处理。
多模态大语言模型(MLLM)通过融合文本与视觉输入拓展真实应用,但其广泛知识引发隐私泄露、毒性内容及知识产权问题。机器删忆(MU)可选择性遗忘特定知识同时保持模型整体性能。现有基于神经元编辑的删忆方法在多模态场景面临两大挑战:(1)因点级归因无法捕捉跨模态层间信息流,导致模态间遗忘不一致;(2)删除敏感神经元时可能破坏重要推理路径,损害通用性能。为此,我们提出多模态影响力神经路径编辑器(MIP-Editor),引入模态专属归因分数识别负责遗忘集知识的关键神经路径,并通过表征误导实现影响路径感知的神经元编辑。该策略实现跨模态协同遗忘,同时保留模型通用能力。实验表明,MIP-Editor在多模态任务中最大遗忘率达87.75%,通用知识保留提升54.26%;在文本任务中遗忘率达80.65%,通用性能保留77.9%。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) extend foundation models to real-world applications by integrating inputs such as text and vision. However, their broad knowledge capacity raises growing concerns about privacy leakage, toxicity mitigation, and intellectual property violations. Machine Unlearning (MU) offers a practical solution by selectively forgetting targeted knowledge while preserving overall model utility. When applied to MLLMs, existing neuron-editing-based MU approaches face two fundamental challenges: (1) forgetting becomes inconsistent across modalities because existing point-wise attribution methods fail to capture the structured, layer-by-layer information flow that connects different modalities; and (2) general knowledge performance declines when sensitive neurons that also support important reasoning paths are pruned, as this disrupts the model's ability to generalize. To alleviate these limitations, we propose a multimodal influential neuron path editor (MIP-Editor) for MU. Our approach introduces modality-specific attribution scores to identify influential neuron paths responsible for encoding forget-set knowledge and applies influential-path-aware neuron-editing via representation misdirection. This strategy also enables effective and coordinated forgetting across modalities while preserving the model's general capabilities. Experimental results demonstrate that MIP-Editor achieves a superior unlearning performance on multimodal tasks, with a maximum forgetting rate of 87.75% and up to 54.26% improvement in general knowledge retention. On textual tasks, MIP-Editor achieves up to 80.65% forgetting and preserves 77.9% of general performance. Codes are available at https://github.com/PreckLi/MIP-Editor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。