提出模块感知的高效机器遗忘方法，精准定位并移除关键参数。
Module-Aware Parameter-Efficient Machine Unlearning on Transformers
- 通过可学习掩码识别注意力头与卷积滤波器中的关键参数
- 在多个Transformer模型上实现优于现有方法的遗忘效果
- 适合需要满足隐私合规的大型模型更新场景
Transformer已成为众多预训练大模型的基础，在各类应用中取得显著成功。机器遗忘旨在高效消除特定数据的影响以符合隐私法规，其核心是限制对影响关键参数的更新。然而，现有参数高效的遗忘方法多为模块无感知设计，容易误判关键参数，导致Transformer遗忘性能下降。本文提出{ t MAPE-Unlearn}，一种模块感知的参数高效机器遗忘方法，利用可学习的一对掩码精准定位Transformer中注意力头和滤波器内的影响关键参数。掩码的学习目标基于遗忘需求设计，并通过带有热启动的贪心搜索算法高效优化。在多种Transformer模型与数据集上的大量实验表明，{ t MAPE-Unlearn}在遗忘效果与鲁棒性方面均表现优异。
原文摘要 · Abstract (English)
Transformer has become fundamental to a vast series of pre-trained large models that have achieved remarkable success across diverse applications. Machine unlearning, which focuses on efficiently removing specific data influences to comply with privacy regulations, shows promise in restricting updates to influence-critical parameters. However, existing parameter-efficient unlearning methods are largely devised in a module-oblivious manner, which tends to inaccurately identify these parameters and leads to inferior unlearning performance for Transformers. In this paper, we propose {\tt MAPE-Unlearn}, a module-aware parameter-efficient machine unlearning approach that uses a learnable pair of masks to pinpoint influence-critical parameters in the heads and filters of Transformers. The learning objective of these masks is derived by desiderata of unlearning and optimized through an efficient algorithm featured by a greedy search with a warm start. Extensive experiments on various Transformer models and datasets demonstrate the effectiveness and robustness of {\tt MAPE-Unlearn} for unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。