针对多模态大模型遗忘难题,提出按模态感知剪枝新方法。
Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models
- 按模态识别对敏感信息贡献大的神经元,分阶段选择并剪枝。
- 在不显著影响模型整体能力的前提下,实现各模态的均衡遗忘。
- 适合需要隐私保护的多模态大模型应用场景。
生成式模型如大型语言模型(LLMs)和多模态大语言模型(MLLMs)在海量数据上训练后可能记忆并无意泄露敏感信息,引发伦理与隐私问题。尽管已有研究探讨了LLM中的此类问题,但因多模态知识跨模态纠缠,MLLM的全面遗忘更具挑战性。为此,我们提出模态感知神经元遗忘(MANU)框架,通过识别并剪除对目标遗忘数据具有相对重要性的神经元,实现针对不同模态的精准遗忘。MANU包含两个阶段:重要神经元选择与选择性剪枝。第一阶段识别跨模态中对目标遗忘知识最具影响力的神经元,第二阶段则剪除这些神经元。该方法能有效隔离并移除各模态中贡献最大的敏感信息神经元,同时保持保留知识的完整性。在多种MLLM架构上的实验表明,MANU可在不显著影响模型整体性能的前提下,实现各模态更平衡、更全面的遗忘。
原文摘要 · Abstract (English)
Generative models such as Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) trained on massive datasets can lead them to memorize and inadvertently reveal sensitive information, raising ethical and privacy concerns. While some prior works have explored this issue in the context of LLMs, it presents a unique challenge for MLLMs due to the entangled nature of knowledge across modalities, making comprehensive unlearning more difficult. To address this challenge, we propose Modality Aware Neuron Unlearning (MANU), a novel unlearning framework for MLLMs designed to selectively clip neurons based on their relative importance to the targeted forget data, curated for different modalities. Specifically, MANU consists of two stages: important neuron selection and selective pruning. The first stage identifies and collects the most influential neurons across modalities relative to the targeted forget knowledge, while the second stage is dedicated to pruning those selected neurons. MANU effectively isolates and removes the neurons that contribute most to the forget data within each modality, while preserving the integrity of retained knowledge. Our experiments conducted across various MLLM architectures illustrate that MANU can achieve a more balanced and comprehensive unlearning in each modality without largely affecting the overall model utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。