让大模型原生支持按数据源删除知识,不重训也能精准去除非必要内容。
Natively Unlearnable Large Language Models

- 用共享主干+稀疏激活的专用节点,自然分离各数据源信息
- 禁用特定节点即可删除单篇文章知识,效果接近从头训练
- 适合需要数据可控、可删除的AI应用,如隐私保护或合规场景
去学习旨在消除特定训练数据的影响,但因不同数据源贡献纠缠难以实现。我们提出NULLs(原生不可学习大模型),在保持跨源联合学习的同时,将各源信息隔离于独立参数。通过训练共享主干神经元与稀疏激活的专用存储池,源特异性信息自然集中于对应池,共性信息则汇聚于主干。部署时仅需关闭某源对应的存储池即可实现去学习,无需梯度更新或原始数据。实验表明,该方法可扩展至约600万篇维基百科文章,每篇作为独立源处理;删除一篇文档仅移除其专属知识,保留与语义相关文档的共享事实,效果接近从头训练。案例研究显示,对《哈利·波特》系列书籍的去学习具有鲁棒性,能抵抗对抗性提取和反向复现。此外,模型在下游任务上表现与标准Transformer相当。结果表明,源级去学习可嵌入模型训练过程,无需牺牲共享表征优势。
原文摘要 · Abstract (English)
Unlearning aims to remove the influence of specific training data sources, but this has proved challenging because the contributions of different sources are entangled within the model. Isolating source contributions to disjoint parameters makes removal easier, though it obstructs joint learning across sources. We propose NULLs (Natively Unlearnable LLMs), a model class that satisfies the two opposing goals of isolating source-specific contributions and learning jointly across sources, by training a set of shared backbone neurons alongside a pool of sparsely activated sinks. During training, information specific to a source naturally concentrates in its sinks while information shared across sources accumulates in the backbone. A source is then unlearned at deployment by disabling its corresponding sinks, with no gradient updates and no access to the retained data. We show that NULLs scales to Wikipedia's ~6M articles, isolating each as an independent source. Unlearning a single article removes knowledge specific to it while preserving facts shared with semantically related articles, closely matching retraining from scratch. We note that unlearning with NULLs is also robust: in a case study of unlearning the Harry Potter books, NULLs resists both adversarial extraction and relearning that reverses post-hoc unlearning. Finally, NULLs preserves general language capabilities, matching a standard transformer on downstream benchmarks. Together, these results suggest that source-level unlearning need not be an afterthought. It can be built natively into LLM training while retaining the benefits of shared representation learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。