提出一种更稳定高效的LLM遗忘方法，解决传统方法易崩溃、难调控的问题。
BLADE: Bilevel Low-rank Augmented-Lagrangian Erasure for LLM Unlearning
- 用分层优化框架控制遗忘过程，确保模型保持连贯性。
- 在多个数据集上提升性能，最高增益达9%，且支持多次遗忘操作。
- 适合需要反复擦除敏感信息的高可靠性场景使用。
现有大模型遗忘方法存在鲁棒性问题：遗忘损失无界会破坏模型连贯性，固定权重难以适应保留难度变化，且单一基准有效的方法在规模扩展或重复应用时失效。本文提出BLADE，一种受约束的双层优化框架，包含三项机制：受限熵遗忘损失，在令牌达到足够不确定性后梯度恒为零；非对称增广拉格朗日项，一旦违反即永久加强保留保护；以及仅作用于LoRA适配器的双层结构，可在每次遗忘前修复保留损伤。BLADE在三个基准族中全面领先，相较最强基线在TOFU上提升6%、MUSE Books上提升9%、KnowUndo上提升7%；在MUSE News上支持4倍规模扩展与4次连续遗忘步骤，而最优对比方法完全崩溃。
原文摘要 · Abstract (English)
Existing LLM unlearning methods struggle with robustness: unbounded forget losses degrade model coherence, fixed-weight balancing cannot adapt as retain difficulty shifts mid-training, and methods that work on one benchmark falter under scaling or repeated application. We propose BLADE, a constrained bilevel framework whose three mechanisms give smooth, predictable control over the optimization landscape: a clamped-entropy forget loss whose gradient is exactly zero once a token reaches sufficient uncertainty; an asymmetric augmented Lagrangian that permanently ratchets retain protection after any violation; and a bilevel structure confined to LoRA adapters that repairs retain damage before each forgetting step. BLADE dominates across three benchmark families, improving average composite scores over the strongest baselines by $6$% on TOFU, $9$% on MUSE Books, and $7$% on KnowUndo, and it remains stable under $4\times$ scaling and $4$ sequential unlearning steps on MUSE News where the best competing method collapses entirely.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。