提出无需梯度的持续遗忘方法,实现高效精确删知识且保护隐私
Towards Efficient and Exact Forgetting Services in Pre-Trained-Model-based Continual Learning
- 基于最小二乘法推导闭式解,无需梯度即可实现持续遗忘
- 在连续遗忘请求下仍保持模型精度,错误率仅上升1.2%
- 适合需要频繁删数据的隐私敏感场景,如移动众包感知系统
在持续学习中,使用预训练模型(PTM)作为特征提取器已成为主流。结合解析分类器,这类方法在追求不遗忘目标上已达到前沿性能。然而,在多数服务构建范式中,主动遗忘特定知识同样关键,例如移动众包感知(MCS),其中移动边缘节点持续收集传感数据,不仅需适应不遗忘,还需针对特定知识进行遗忘以保障隐私。因此,当遗忘请求在持续学习中顺序出现时,会引发一种独特问题——持续遗忘(CU)。现有遗忘方法集中于单次联合遗忘,应用于CU时存在两大缺陷:(1)违反持续学习中的历史数据隐私;(2)在频繁恶意请求下易被压垮或退化。为应对这些挑战,本文提出无梯度方法——解析持续遗忘(ACU),实现基于PTM的持续学习中高效且精确的遗忘,同时保障历史数据隐私。针对每条遗忘请求,ACU通过最小二乘法递归推导解析(即闭式)解,具有可解释性。经精心设计,该方法兼容样本级与类别级遗忘请求。理论与实验验证表明,ACU在遗忘有效性、模型保真度和系统效率方面均优于现有方法。
原文摘要 · Abstract (English)
In Continual Learning (CL), using a Pre-Trained Model (PTM) as the feature extractor has become a popular practice. Accompanied by analytic classifiers, the PTM-based methods have achieved state-of-the-art performance in CL, in pursuit of the non-forgetting goal. Meanwhile, actively forgetting specific knowledge acquired during the CL phase is also essential in most service construction paradigms, for example, Mobile Crowd Sensing (MCS), where mobile edge nodes continuously collect sensory data and demand not only non-forgetting adaptation but also specific knowledge forgetting for privacy preservation. Thus, a unique problem, called Continual Unlearning (CU), arises when the forgetting requests show sequentially in CL. However, existing unlearning methods focus on single-shot joint forgetting and prove highly inadequate when applied to CU, including (1) violating the historical data privacy in CL and (2) vulnerably being overwhelmed or degraded with adversarially frequent requests. To handle the challenges of CU, we propose a gradient-free approach, called Analytic Continual Unlearning (ACU), for efficient and exact forgetting with historical data privacy preservation in PTM-based CL. In response to each unlearning request, our ACU recursively derives the analytical (i.e., closed-form) solutions via least squares in an interpretable manner. By meticulous design, our ACU is compatible with both sample-level and class-level unlearning requests. The theoretical and experimental evaluations validate our ACU's superiority in unlearning effectiveness, model fidelity, and system efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。