让AI忘记特定图像信息，同时不丢掉回答能力
Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning
- 用视觉线索识别回答中关键词，重点清除敏感信息
- 在两个基准测试上实现强遗忘效果，且回答仍通顺
- 适合需要保护隐私的多模态AI应用
多模态大模型（MLLM）中的去学习（unlearning）旨在防止模型在被问及目标图像时泄露隐私信息。现有方法大多沿用语言模型的策略，对所有回答词汇一视同仁，忽视其在去学习过程中的重要性差异，且仅关注语言模态，忽略能指示关键词汇的视觉线索。本文提出视觉引导的关键词正则化方法（ViKeR），利用无关视觉输入预测去学习后的理想词级分布，并以此指导去学习过程，优先处理关键词汇。通过信息熵定义关键词汇，并借助词级梯度重加权机制强化关键词汇的更新。在MLLMU和CLEAR两个基准上的实验表明，该方法能有效实现去学习，同时缓解遗忘问题并保持回答连贯性。
原文摘要 · Abstract (English)
Unlearning in Multimodal Large Language Models (MLLMs) prevents the model from revealing private information when queried about target images. Existing MLLM unlearning methods largely adopt approaches developed for LLMs. They treat all answer tokens uniformly, disregarding their varying importance in the unlearning process. Moreover, these methods focus exclusively on the language modality, disregarding visual cues that indicate key tokens in answers. In this paper, after formulating the problem of unlearning in multimodal question answering for MLLMs, we propose Visual-Guided Key-Token Regularization (ViKeR). We leverage irrelevant visual inputs to predict ideal post-unlearning token-level distributions and use these distributions to regularize the unlearning process, thereby prioritizing key tokens. Further, we define key tokens in unlearning via information entropy and discuss ViKeR's effectiveness through token-level gradient reweighting, which amplifies updates on key tokens. Experiments on MLLMU and CLEAR benchmarks demonstrate that our method effectively performs unlearning while mitigating forgetting and maintaining response coherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。