让多模态大模型删掉特定图像特征，同时保留文本知识。
MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models
- 用几何约束梯度上升法，仅删除目标实体的视觉模式。
- 在多项评测中优于直接微调或负向偏好优化基线方法。
- 适合需要隐私保护又想保留语言理解能力的研究者。
多模态大语言模型（MLLM）中的机器可撤销性（MU）尚处起步阶段。本文重新定义了该场景下的多模态可撤销任务：仅移除与特定实体相关的视觉模式，同时保留语言模型主干中对应的文本知识。为此，提出新型几何约束梯度上升方法 MMUnlearner，通过联合约束剩余概念与文本知识的权重显著性图，更新模型权重，从而保护非目标知识的关键参数。大量实验表明，MMUnlearner 在所有评估维度上均超越直接使用视觉问答数据进行梯度上升（GA）或负向偏好优化（NPO）微调的基线方法。代码已开源。
原文摘要 · Abstract (English)
Recent progress in Machine Unlearning (MU) has introduced solutions for the selective removal of private or sensitive information encoded within deep neural networks. Nonetheless, MU for Multimodal Large Language Models (MLLMs) remains in its nascent phase. Therefore, we propose to reformulate the task of multimodal MU in the era of MLLMs, which aims to erase only the visual patterns associated with a given entity while preserving the corresponding textual knowledge encoded within the original parameters of the language model backbone. Furthermore, we develop a novel geometry-constrained gradient ascent method MMUnlearner. It updates the weights of MLLMs with a weight saliency map jointly restricted by the remaining concepts and textual knowledge during unlearning, thereby preserving parameters essential for non-target knowledge. Extensive experiments demonstrate that MMUnlearner surpasses baselines that finetuning MLLMs with VQA data directly through Gradient Ascent (GA) or Negative Preference Optimization (NPO), across all evaluation dimensions. Our code can be found in [this URL](https://github.com/Z1zs/MMUnlearner).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。