arXiv:2509.12958cs.AI2025-09

动态分配隐私预算,只删敏感信息,保关键知识。

Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning

  • 按词元语义敏感度动态分配差分隐私预算
  • 在保持旧任务准确率的同时实现强隐私保护
  • 适合需持续学习且重视隐私的场景

持续学习(CL)模型虽能逐步获取知识,但因累积多样信息而面临严重且常被忽视的隐私风险。传统统一差分隐私(DP)预算方法对所有数据一视同仁,导致模型性能显著下降,限制了其在隐私敏感领域的应用。为此,我们提出隐私增强型持续学习框架(PeCL),实现‘遗忘敏感内容,保留关键知识’。首先,提出基于词元级别的动态差分隐私策略,根据单个词元的语义敏感度自适应分配隐私预算,确保对隐私实体的强保护,同时减少对非敏感通用知识的噪声干扰。其次,引入隐私引导的记忆塑形模块,利用动态DP机制分析出的敏感性,智能地从模型记忆和参数中遗忘敏感信息,同时显式保留对缓解灾难性遗忘至关重要的任务不变历史知识。大量实验表明,PeCL在隐私保护与模型效用之间取得更优平衡,相比基线模型,在保持先前任务高准确率的同时实现稳健隐私保障。

原文摘要 · Abstract (English)

Continual Learning (CL) models, while adept at sequential knowledge acquisition, face significant and often overlooked privacy challenges due to accumulating diverse information. Traditional privacy methods, like a uniform Differential Privacy (DP) budget, indiscriminately protect all data, leading to substantial model utility degradation and hindering CL deployment in privacy-sensitive areas. To overcome this, we propose a privacy-enhanced continual learning (PeCL) framework that forgets what's sensitive and remembers what matters. Our approach first introduces a token-level dynamic Differential Privacy strategy that adaptively allocates privacy budgets based on the semantic sensitivity of individual tokens. This ensures robust protection for private entities while minimizing noise injection for non-sensitive, general knowledge. Second, we integrate a privacy-guided memory sculpting module. This module leverages the sensitivity analysis from our dynamic DP mechanism to intelligently forget sensitive information from the model's memory and parameters, while explicitly preserving the task-invariant historical knowledge crucial for mitigating catastrophic forgetting. Extensive experiments show that PeCL achieves a superior balance between privacy preserving and model utility, outperforming baseline models by maintaining high accuracy on previous tasks while ensuring robust privacy.

持续学习差分隐私记忆塑形

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。