提出BID-LoRA框架,实现持续学习与遗忘的统一,仅更新5%参数即可精准删除旧知识。
BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning
- 设计双向低秩适配器,分路径保留、新增和遗忘信息,专注注意力层优化。
- 在CIFAR-100上多轮适应后仍保持高准确率,有效防止知识泄露与退化。
- 适用于人脸管理等需动态增删用户的实际场景,适合隐私保护需求高的应用。
深度学习的发展要求系统既能通过持续学习(CL)获取新知识,又能通过机器遗忘(MU)移除过时、敏感或私密信息。然而,尽管CL方法已较成熟,MU技术仍处于初级阶段,导致兼具二者能力的统一框架存在关键缺口。我们发现,简单结合现有CL与MU方法会导致知识泄露,并引发多次适应循环中基础知识的渐进退化。为此,我们正式提出持续学习与遗忘(CLU)统一范式,包含三个核心目标:(i)精确删除不需要的知识,(ii)高效整合新知识并保留原有信息,(iii)最小化各轮次间的知识泄露。我们提出双向低秩适配(BID-LoRA),在注意力层引入三种专用适配路径——保留、新增、遗忘,并结合逃逸遗忘机制,将待遗忘类嵌入推向与保留知识最远的位置,仅更新5%参数。在CIFAR-100上的实验表明,BID-LoRA在多轮适应中优于基线模型。进一步在CASIA-Face100(一个精选的人脸识别子集)上验证,证明其在真实身份管理系统中支持用户动态注册与撤回的实用性。
原文摘要 · Abstract (English)
Recent advances in deep learning underscore the need for systems that can not only acquire new knowledge through Continual Learning (CL) but also remove outdated, sensitive, or private information through Machine Unlearning (MU). However, while CL methods are well-developed, MU techniques remain in early stages, creating a critical gap for unified frameworks that depend on both capabilities. We find that naively combining existing CL and MU approaches results in knowledge leakage a gradual degradation of foundational knowledge across repeated adaptation cycles. To address this, we formalize Continual Learning Unlearning (CLU) as a unified paradigm with three key goals: (i) precise deletion of unwanted knowledge, (ii) efficient integration of new knowledge while preserving prior information, and (iii) minimizing knowledge leakage across cycles. We propose Bi-Directional Low-Rank Adaptation (BID-LoRA), a novel framework featuring three dedicated adapter pathways-retain, new, and unlearn applied to attention layers, combined with escape unlearning that pushes forget-class embeddings to positions maximally distant from retained knowledge, updating only 5% of parameters. Experiments on CIFAR-100 show that BID-LoRA outperforms CLU baselines across multiple adaptation cycles. We further evaluate on CASIA-Face100, a curated face recognition subset, demonstrating practical applicability to real-world identity management systems where new users must be enrolled and withdrawn users removed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。